Methods of obtaining panoramic images using rotationally symmetric wide-angle lenses and devices thereof

ABSTRACT

The present invention provides methods of obtaining panoramic images that appear most natural to the naked eye by executing a mathematically precise image processing operation on a wide-angle image acquired using a wide-angle lens that is rotationally symmetric about an optical axis, and devices using the methods. Imaging systems using this method can be used not only in security surveillance applications for indoor and outdoor environments, but also in diverse areas such as video phones for apartment entrance doors, rear view cameras for vehicles, visual sensors for unmanned aerial vehicles and robots, and broadcasting cameras. Also, it can be used to obtain panoramic photographs using digital cameras.

TECHNICAL FIELD

The present invention generally relates to mathematically precise imageprocessing methods of extracting panoramic images, which appear mostnatural to the naked eye, from images acquired using a camera equippedwith a wide-angle lens that is rotationally symmetric about an opticalaxis, as well as devices using the methods.

BACKGROUND ART

Panoramic camera, which captures the 360° view of scenic places such astourist resorts, is an example of a panoramic imaging system. Panoramicimaging system is an imaging system that captures the views one couldget by making one complete turn-around from a given spot. On the otherhand, omnidirectional imaging system captures the view of every possibledirection from a given spot. Omnidirectional imaging system provides aview that a person could observe from a given position by turning aroundas well as looking up and down. In a mathematical terminology, the solidangle of the region that can be captured by the imaging system is 4πsteradian.

There have been a lot of studies and developments of panoramic imagingsystems not only in the traditional areas such as photographingbuildings, nature scenes, and heavenly bodies, but also insecurity/surveillance systems using CCD (charge-coupled device) or CMOS(complementary metal-oxide-semiconductor) cameras, virtual touring ofreal estates, hotels and tourist resorts, and navigational aids formobile robots and unmanned aerial vehicles (UAV).

As a viable method of obtaining panoramic images, people are activelyresearching on catadioptric panoramic imaging systems, which are imagingsystems employing both mirrors and refractive lenses. Shown in FIG. 1 isa schematic diagram of a general catadioptric panoramic imaging system.As schematically shown in FIG. 1, a catadioptric panoramic imagingsystem(100) of prior arts includes as constituent elements arotationally symmetric panoramic mirror(111), of which thecross-sectional profile is close to an hyperbola, a lens(112) that islocated on the rotational-symmetry axis(101) of the mirror(111) andoriented toward the said mirror(111), and an image acquisition meansthat includes a camera body(114) having an image sensor(113) inside.Then, an incident ray(105) having an altitude angle δ, which originatesfrom every 360° directions around the mirror and propagates toward therotational-symmetry axis(101), is reflected on a point M on the mirrorsurface(111), and captured by the image sensor(113) as a reflectedray(106) having a zenith angle θ with respect to the rotational-symmetryaxis(101). Here, the altitude angle refers to an angle measured from theground plane (i.e., X-Y plane) toward the zenith. FIG. 2 is a conceptualdrawing of an exemplary rural landscape obtainable using thecatadioptric panoramic imaging system(100) of prior art schematicallyshown in FIG. 1. As illustrated in FIG. 2, a photographic film or animage sensor(213) has a square or a rectangular shape, while a panoramicimage(233) obtained using a panoramic imaging system(100) has an annularshape. Non-hatched region in FIG. 2 constitutes the panoramic image, andthe hatched circle in the center corresponds to the area at the backsideof the camera, which is not captured because the camera body occludesits view. Within this circle lies the image of the camera itselfreflected by the mirror(111). On the other hand, the hatched regions atthe four corners originate from the fact that the diagonal field of viewof the camera lens(112) is larger than the field of view of thepanoramic mirror(111). In this region lies the image of the area at thefront side of the camera observable in absence of the panoramic mirror.FIG. 3 is an exemplary unwrapped panoramic image(334) obtained from thering-shaped panoramic image(233) by cutting along the cutting-line(233c) and converting into a perspectively normal view using an imageprocessing software.

For the unwrapped panoramic image(334) in FIG. 3 to appear natural tothe naked eye, the raw panoramic image(233) prior to the unwrappingoperation must be captured by a panoramic lens following a certainprojection scheme. Here, a panoramic lens refers to a complex lenscomprised of a panoramic mirror(111) and a refractive lens(112). FIG. 4is a conceptual drawing of an object plane(431) employed in a panoramicimaging system following a rectilinear projection scheme, and FIG. 5 isa conceptual drawing of a raw panoramic image(533) obtained by capturingthe scene on the object plane of FIG. 4 using the said panoramic imagingsystem. In such a rectilinear panoramic imaging system, a cylindricalobject plane(131, 431) is assumed, of which the rotational symmetry axiscoincides with the optical axis of the panoramic lens. In FIG. 4, it ispreferable that the rotational symmetry axis of the cylindrical objectplane(431) is perpendicular to the ground plane(417).

Referring to FIG. 1 and FIG. 4, the radius of the said object plane(131)is S, and the panoramic lens comprised of a panoramic mirror(111) and arefractive lens(112) forms the image of the point(104) of an objectlying on the said object plane(131), in other words, the image point P,on the focal plane(132). To obtain a sharp image, the sensor plane(113)of the image sensor must coincide with the said focal plane(132). Aray(106) that arrives at the said image point P is first reflected at apoint M on the panoramic mirror(111) and passes through the nodal pointN of the refractive lens(112). Here, the nodal point is the position ofa pinhole when the camera is approximated as an ideal pinhole camera.The distance from the nodal point to the focal plane(132) isapproximately equal to the effective focal length f of the refractivelens(112). For the simplicity of argument, we will refer the ray(105)before the reflection at the mirror as the incident ray, and the rayafter the reflection as the reflected ray(106). If the reflected ray hasa zenith angle θ with respect to the optical axis(101) of the camera,then, the distance r from the center of the sensor plane(113), in otherwords, the intersection point O between the sensor plane(113) and theoptical axis(101), to the point P on the image sensor plane, whereon thereflected ray(106) is captured, is given by Eq. 1.

r=f tan θ  [Math Figure 1]

For a panoramic lens following a rectilinear projection scheme, theheight in the object plane(131), in other words, the distance Z measuredparallel to the optical axis, is proportional to the distance r on thesensor plane. The axial radius of the point M on the panoramic mirrorsurface(111), whereon the reflection has occurred, is ρ, and the heightis z, and the axial radius of the corresponding point(104) on the objectplane(131) is S, and the height is Z. Since the altitude angle of thesaid incident ray(105) is δ, the height Z of the said object is given byEq. 2.

Z=z+(S−ρ)tan δ  [Math Figure 2]

If the distance from the camera to the object plane is large compared tothe size of the camera (i.e., S>>ρ, Z>>z), then Eq. 2 can beapproximated as Eq. 3.

Z≅S tan δ  [Math Figure 3]

Therefore, if the radius S of the object plane is fixed, then the heightof the object (i.e., the object size) is proportional to tan δ, and theaxial radius of the corresponding image point (i.e., the image size) onthe focal plane is proportional to tan θ. If tan δ is proportional totan θ in this manner, then the image of the object on the object planeis captured on the image sensor with its vertical proportions preserved.Incidentally, referring to FIG. 1, it can be noticed that both thealtitude angle of the incident ray and the zenith angle of the reflectedray have upper bounds and lower bounds. If the range of the altitudeangle of the incident ray is from δ₁ to δ₂ (δ₁≦δ≦δ₂), and the range ofthe zenith angle of the reflected ray is from θ₁ to θ₂ (θ₁≦θ≦θ₂), thenthe range of the corresponding object height on the object plane is fromZ₁=S tan δ₁ to Z₂=S tan δ₂(Z₁≦Z≦Z₂), and the range of the axial radiusof the image point on the focal plane is from r₁=f tan θ₁ to r₂=f tanθ₂(r₁≦r≦r₂). The projection scheme for these r and Z to be in proportionto each other is given by Eq. 4.

$\begin{matrix}{{r(\delta)} = {r_{1} + {\frac{r_{2} - r_{1}}{{\tan \; \delta_{2}} - {\tan \; \delta_{1}}}\left( {{\tan \; \delta} - {\tan \; \delta_{1}}} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Therefore, a most natural panoramic image can be obtained when thepanoramic lens implements the rectilinear projection scheme given by Eq.4. One disadvantage of such a panoramic imaging system is that there areconsiderable numbers of unused pixels in the image sensor. FIG. 6 is aschematic diagram illustrating the degree of pixel utilization on thesensor plane(613) of an image sensor having the standard 4:3 aspectratio. Image sensors with the ratio of the lateral side B to thelongitudinal side V equal to 1:1 or 16:9 are not many in kinds nor cheapin price, and most of the image sensors are manufactured with the ratioof 4:3. Assuming an image sensor having the 4:3 aspect ratio, the areaA₁ of the image sensor plane(613) is given by Eq. 5.

$\begin{matrix}{A_{1} = {{BV} = {\frac{4}{3}V^{2}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 5} \right\rbrack\end{matrix}$

On such an image sensor plane, the panoramic image(633) is formedbetween the outer rim(633 b) and the inner rim(633 a) of an annularregion, where the two rims constitute concentric circles. Here, the saidsensor plane(613) coincides with a part of the focal plane(632) of thelen, and the said panoramic image(633) exists on a part of the sensorplane(613). In FIG. 6, the outer radius of the panoramic image(633) isr₂, and the inner radius is r₁. Therefore, the area A₂ of the panoramicimage is given by Eq. 6.

A ₂=π(r ₂ ² −r ₁ ²)  [Math Figure 6]

Referring to FIG. 2 and FIG. 5, the height of the unwrapped panoramicimage is given by the difference between the outer radius and the innerradius, in other words, by r₂−r₁. On the other hand, the lateraldimension of the unwrapped panoramic image is given by 2πr₁ or 2πr₂,depending on which radius is taken as a base. Therefore, the outerradius r₂ and the inner radius r₁ must have an appropriate ratio, and2:1 can be considered as a proper ratio. Furthermore, to make themaximum use of pixels, it is desirable that the outer rim(633 b) of thepanoramic image contacts the lateral sides of the image sensorplane(613). Therefore, it is preferable that r₂=(½)V, and r₁=(½)r₂=(¼)V.Under these conditions, the area of the panoramic image(633) is given byEq. 7.

$\begin{matrix}{A_{2} = {{\pi \left\{ {\left( {\frac{1}{2}V} \right)^{2} - \left( {\frac{1}{4}V} \right)^{2}} \right\}} = {\frac{3\pi}{16}V^{2}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 7} \right\rbrack\end{matrix}$

Therefore, the ratio between the area A₂ of the panoramic image(633) andthe area A₁ of the image sensor plane(613) is given by Eq. 8.

$\begin{matrix}{\frac{A_{2}}{A_{1}} = {\frac{\frac{3\pi}{16}V^{2}}{\frac{4}{2}V^{2}} = {\frac{9\pi}{64} \cong 0.442}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 8} \right\rbrack\end{matrix}$

Thus, the percentage of pixel utilization is less than 50%, and thepanoramic imaging systems of prior arts have a disadvantage in thatpixels are not efficiently used.

Another method of obtaining a panoramic image is to employ a fisheyelens with a wide field of view (FOV). For example, the entire sky andthe horizon can be captured in a single image by pointing a cameraequipped with a fisheye lens with 180° FOV toward the zenith (i.e., theoptical axis of the camera is aligned perpendicular to the groundplane). On this reason, fisheye lenses have been often referred to as“all-sky lenses”. Particularly, a high-end fisheye lens by Nikon,namely, 6 mm f/5.6 Fisheye-Nikkor, has a FOV of 220°. Therefore, acamera mounted with this lens can capture even a portion of the backsideof the camera as well as the front side of the camera. Then, a panoramicimage can be obtained from thus obtained fisheye image by the samemethods as illustrated in FIG. 2 and FIG. 3.

In many cases, imaging systems are installed on vertical walls. Imagingsystems installed on the outside walls of a building for the purpose ofmonitoring the surroundings, or a rear view camera for monitoring thebackside of a passenger car are such examples. In such cases, it isinefficient if the horizontal field of view is significantly larger than180°. This is because a wall, which is not needed to be monitored, takesup a large space in the monitor screen, pixels are wasted, and screenappears dull. Therefore, a horizontal FOV around 180° is moreappropriate for such cases. Nevertheless, a fisheye lens with 180°FOV isnot desirable for such application. This is because the barreldistortion, which accompanies a fisheye lens, evokes psychologicaldiscomfort and abhorred by the consumer.

An example of an imaging system, which can be installed on an interiorwall for the purpose of monitoring the entire room, is given by apan•tilt•zoom camera. Such a camera is comprised of a video camera,which is equipped with an optical zoom lens, mounted on a pan•tiltstage. Pan is an operation of rotating in the horizontal direction for agiven angle, and tilt is an operation of rotating in the verticaldirection for a given angle. In other words, if we assume that thecamera is at the center of a celestial sphere, then pan is an operationof changing the longitude, and tilt is an operation of changing thelatitude. Therefore, the theoretical range of pan operation is 360°, andthe theoretical range of tilt operation is 180°. The shortcomings of apan•tilt•zoom camera include high price, large size and heavy weight.Optical zoom lens is large, heavy and expensive due to the difficulty indesign and the complicated structure. Also, a pan•tilt stage is anexpensive device not cheaper than a camera. Therefore, it cost aconsiderable sum of money to install a pan•tilt•zoom camera.Furthermore, since a pan•tilt•zoom camera is large and heavy, this factcan become a serious impediment to certain applications. Examples ofsuch cases include airplanes where the weight of the payload is ofcritical importance, or when a strict size limitation exists in order toinstall a camera in a confined space. Furthermore, pan•tilt•zoomoperation takes a time because it is a mechanical operation. Therefore,depending on the particular application at hand, such a mechanicalresponse may not be fast enough.

References 1 and 2 provide fundamental technologies of extracting animage having a particular viewpoint or projection scheme from an imagehaving other than the desirable viewpoint or projection scheme.Specifically, reference 2 provides an example of a cubic panorama. Inshort, a cubic panorama is a special technique of illustration whereinthe observer is assumed to be located at the very center of an imaginarycubic room made of glass, and the outside view from the center of theglass room is directly transcribed on the region of the glass wallwhereon the ray vector from the object to the observer meets the glasswall. Furthermore, an example of a more advanced technology is providedin the above reference, wherewith reflections from an arbitrarily shapedmirrored surface can be calculated. Specifically, the author ofreference 2 created an imaginary lizard having a highly reflectivemirror-like skin as if made of a metal surface, then set-up anobserver's viewpoint separated from the lizard, and calculated the viewof the imaginary environment reflected on the lizard skin from theviewpoint of the imaginary observer. However, the environment was not areal environment captured by an optical lens, but a computer-createdimaginary environment captured with an imaginary distortion-free pinholecamera.

On the other hand, an imaging system is described in reference 3 that isable to perform pan•tilt•zoom operations without a physically movingpart. The said invention uses a camera equipped with a fisheye lens withmore than 180° FOV in order to take a picture of the environment. Then,the user designates a principal direction of vision using variousdevices such as a joystick, upon which, the computer extracts arectilinear image from the fisheye image that could be obtained byheading a distortion-free camera to that particular direction. The maindifference between this invention and the prior arts is that thisinvention creates a rectilinear image corresponding to the particulardirection the user has designated using devices such as a joystick or acomputer mouse. Such a technology is essential in the field of virtualreality, or when it is desirable to replace mechanical pan•tilt•zoomcamera, and the keyword is “interactive picture”. In this technology,there are no physically moving parts in the camera. As a consequence,the system response is fast, and there is less chance of mechanicalfailure.

Ordinarily, when an imaging system such as a security camera isinstalled, a cautionary measure is taken so that vertical linesperpendicular to the horizontal plane also appear vertical in theacquired image. In such a case, vertical lines still appear verticaleven as mechanical pan•tilt•zoom operation is performed. On the otherhand, in the said invention, vertical lines generally do not appear asvertical lines after software pan•tilt•zoom operation has beenperformed. To remedy such an unnatural result, a rotate operation isadditionally performed, which is not found in a mechanical pan•tilt•zoomcamera. Furthermore, the said invention does not provide the exactamount of rotate angle that is needed in order to display vertical linesas vertical lines. Therefore, the exact rotation angle must be found ina trial-and-error method in order to display vertical lines as verticallines.

Furthermore, the said invention assumes that the projection scheme ofthe fisheye lens is an ideal equidistance projection scheme. But, thereal projection scheme of a fisheye lens generally shows a considerabledeviation from an ideal equidistance projection scheme. Since the saidinvention does not take into account the distortion characteristics of areal lens, images obtained after image processing still showsdistortion.

The invention described in reference 4 remedies the shortcoming of theinvention described in reference 3, namely the inability of taking intoaccount the real projection scheme of a fisheye lens used in imageprocessing. Nevertheless, the defect of not showing vertical lines asvertical lines in the monitor screen has not been resolved.

From another point of view, all animals and plants including human arebound on the surface of the earth due to the gravitational pull, andmost of the events, which need attention or cautionary measure, takeplace near the horizon. Therefore, even though it is necessary tomonitor every 360° direction on the horizon, it is not as important tomonitor high along the vertical direction, for example, as high as tothe zenith or deep down to the nadir. Distortion is unavoidable if wewant to describe the scene of every 360° direction on a two-dimensionalplane. Similar difficulty exists in the cartography where geography onearth, which is a structure on the surface of a sphere, needs to bemapped on a planar two-dimensional atlas. Among all the distortions, thedistortion that appears most unnatural to the people is the distortionwhere vertical lines appear as curved lines. Therefore, even if otherkinds of distortions are present, it is important to make sure that sucha distortion is absent.

Described in reference 5 are the well-known map projection schemes amongthe diverse map projection schemes such as equi-rectangular projection,Mercator projection and cylindrical projection schemes, and reference 6provides a brief history of diverse map projection schemes. Among these,the equi-rectangular projection scheme is the projection scheme mostfamiliar to us when we describe the geography on the earth, or when wedraw the celestial sphere in order to make a map of the constellation.

Referring to FIG. 7, if we assume the surface of the earth is aspherical surface with a radius S, then an arbitrary point Q on theearth's surface has a longitude ψ and a latitude δ. On the other hand,FIG. 8 is a schematic diagram of a planar map drawn according to theequi-rectangular projection scheme. A point Q on the earth's surfacehaving a longitude ψ and a latitude δ has a corresponding point P on theplanar map(834) drawn according to the equi-rectangular projectionscheme. The rectangular coordinate of this corresponding point is givenas (x, y). Furthermore, the reference point on the equator having alongitude 0° and a latitude 0° has a corresponding point O on the planarmap, and this corresponding point O is the origin of the rectangularcoordinate system. Here, according to the equi-rectangular projectionscheme, the same interval in the longitude (i.e., the same angulardistance along the equator) corresponds to the same lateral interval onthe planar map. In other words, the lateral coordinate x on the planarmap(834) is proportional to the longitude.

x=cψ  [Math Figure 9]

Here, c is proportionality constant. Also, the longitudinal coordinate yis proportional to the latitude, and has the same proportionalityconstant as the lateral coordinate.

y=cδ  [Math Figure 10]

The span of the longitude is 360° ranging from −180° to +180°, and thespan of the latitude is 180° ranging from −90° to +90°. Therefore, a mapdrawn according to the equi-rectangular projection scheme must have awidth W:height H ratio of 360:180=2:1. Furthermore, if theproportionality constant c is given as the radius S of the earth, thenthe width of the said planar map is given as the perimeter of the earthmeasured along the equator as given in Eq. 11.

W=2πS  [Math Figure 11]

Such an equi-rectangular projection scheme appears as a naturalprojection scheme considering the fact that the earth's surface is closeto the surface of a sphere. Nevertheless, it is disadvantageous in thatthe size of a geographical area is greatly distorted. For example, twovery close points near the North Pole can appear as if they are on theopposite sides of the earth in a map drawn according to theequi-rectangular projection scheme.

On the other hand, in a map drawn according to the Mercator projectionscheme, the longitudinal coordinate is given as a complex function givenin Eq. 12.

$\begin{matrix}{y = {c\; \ln \left\{ {\tan \left( {\frac{\pi}{4} + \frac{\delta}{2}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 12} \right\rbrack\end{matrix}$

On the other hand, FIG. 9 is a conceptual drawing of a cylindricalprojection scheme or a panoramic perspective. In a cylindricalprojection scheme, an imaginary observer is located at the center N of acelestial sphere(931) with a radius S, and it is desired to make a mapof the celestial sphere centered on the observer, the map covering mostof the region excluding the zenith and the nadir. In other words, thespan of the longitude must be 360° ranging from −180° to +180°, but therange of the latitude can be narrower including the equator within itsspan. Specifically, the span of the latitude can be assumed as rangingfrom −Δ to +Δ, where Δ must be smaller than 90°.

In this projection scheme, a hypothetical cylindrical plane(934) isassumed which contacts the celestial sphere at the equator(903). Then,for a point Q(ψ, δ) on the celestial sphere(931) having a givenlongitude ψ and a latitude δ, a line segment connecting the center ofthe celestial sphere and the point Q is extended until it meets the saidcylindrical plane. This intersection point is designated as P(ψ, δ). Inthis manner, the corresponding point P on the cylindrical plane(934) canbe obtained for every point Q on the celestial sphere(931) within thesaid latitude range. Then, a map having a cylindrical projection schemeis obtained by cutting the cylindrical plane and laying flat on a planarsurface. Therefore, the lateral coordinate of the point P on theflattened-out cylindrical plane is given by Eq. 13, and the longitudinalcoordinate y is given by Eq. 14.

x=Sψ  [Math Figure 13]

y=S tan δ  [Math Figure 14]

Such a cylindrical projection scheme is the natural projection schemefor a panoramic camera that produces a panoramic image by rotating inthe horizontal plane. Especially, if the lens mounted on the rotatingpanoramic camera is a distortion-free rectilinear lens, then theresulting panoramic image exactly follows a cylindrical projectionscheme. In principle, such a cylindrical projection scheme is the mostaccurate panoramic projection scheme. However, the panoramic imageappears unnatural when the latitudinal range is large, and thus it isnot widely used in practice.

Unwrapped panoramic image thus produced and having a cylindricalprojection scheme has a lateral width W given by Eq. 11. On the otherhand, if the range of the latitude is from δ₁ to δ₂, then thelongitudinal height of the unwrapped panoramic image is given by Eq. 15.

H=S(tan δ₂−tan δ₁)  [Math Figure 15]

Therefore, the following equation can be derived from Eq. 11 and Eq. 15.

$\begin{matrix}{\frac{W}{H} = \frac{2\pi}{{\tan \; \delta_{2}} - {\tan \; \delta_{1}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 16} \right\rbrack\end{matrix}$

Therefore, an unwrapped panoramic image following a cylindricalprojection scheme must satisfy Eq. 16.

FIG. 10 is an example of an unwrapped panoramic image given in reference7, and FIG. 11 is an example of an unwrapped panoramic image given inreference 8. FIGS. 10 and 11 have been acquired using panoramic lensesfollowing rectilinear projection schemes, or in the terminology ofcartography, using panoramic lenses following cylindrical projectionschemes. Therefore, in the panoramic images of FIG. 10 and FIG. 11, thelongitudinal coordinate y is proportional to tan δ. On the other hand,by the structure of panoramic lenses, the lateral coordinate x isproportional to the longitude ψ. Therefore, except for theproportionality constant, Eqs. 13 and 14 are satisfied.

In the example of FIG. 10, the lateral size is 2192 pixels, and thelongitudinal size is 440 pixels. Therefore, 4.98 is obtained bycalculating the LHS (left hand side) of Eq. 16. In FIG. 10, the range ofthe vertical incidence angle is from δ₁=−70° to δ₂=50°. Therefore, 1.60is obtained by calculating the RHS (right hand side) of Eq. 16. Thus,the exemplary panoramic image in FIG. 10 does not satisfy theproportionality relation given by Eq. 16. On the other hand, in theexample of FIG. 11, the lateral size is 2880 pixels, and thelongitudinal size is 433 pixels. Therefore, 6.65 is obtained bycalculating the LHS of Eq. 16. In FIG. 11, the range of the verticalincidence angle is from δ₁=−23° to δ₂=23°. Therefore, 7.40 is obtainedby calculating the RHS of Eq. 16. Thus, although the error may be lessthan that of FIG. 10, still the exemplary panoramic image in FIG. 11does not satisfy the proportionality relation given by Eq. 16.

It can be noticed that the unwrapped panoramic images given in FIG. 10and FIG. 11 appear as natural panoramic images despite the fact that thepanoramic images do not satisfy such a proportionality relation. This isbecause of the fact that in a panoramic image, the phenomenon of a linevertical to the ground plane (i.e., a vertical line) appearing as acurved line or as a slanted line is easily noticeable and causes viewerdiscomfort, but the phenomenon of the lateral and the vertical scalesnot matching to each other is not unpleasant to the eye in the samedegree, because a reference for comparing the horizontal and thevertical directions does not usually exist in the environment around thecamera.

All the animals, plants and inanimate objects such as buildings on theearth are under the influence of gravity, and the direction ofgravitational force is the up-right direction or the vertical direction.Ground plane is fairly perpendicular to the gravitational force, butneedless to say, it is not so on a slanted ground. Therefore, the word“ground plane” actually refers to the horizontal plane, and the verticaldirection is the direction perpendicular to the horizontal plane. Evenif we refer them as the ground plane, the lateral direction, and thelongitudinal direction, for the sake of simplicity in argument, theground plane must be understood as the horizontal plane, the verticaldirection must be understood as the direction perpendicular to thehorizontal plane, and the horizontal direction must be understood as adirection parallel to the horizontal plane, whenever an exact meaning ofa term needs to be clarified.

Panoramic lenses described in references 7 and 8 take panoramic imagesin one shot with the optical axes of the panoramic lenses alignedvertical to the ground plane. Incidentally, a cheaper alternative to thepanoramic image acquisition method by the previously described camerawith a horizontally-rotating lens consist of taking an image with anordinary camera with the optical axis horizontally aligned, andrepeating to take pictures after horizontally rotating the optical axisby a certain amount. Four to eight pictures are taken in this way, and apanoramic image with a cylindrical projection scheme can be obtained byseamlessly joining the pictures consecutively. Such a technique iscalled stitching. QuickTime VR from Apple computer inc. is commercialsoftware supporting this stitching technology. This method requires acomplex, time-consuming, and elaborate operation of precisely joiningseveral pictures and correcting the lens distortion.

According to the reference 9, another method of obtaining a panoramic oran omnidirectional image is to take a hemispherical image byhorizontally pointing a camera equipped with a fisheye lens with morethan 180° FOV, and then point the camera to the exact opposite directionand take another hemispherical image. By stitching the two imagesacquired by the camera using appropriate software, one omnidirectionalimage having the views of every direction (i.e., 4π steradian) can beobtained. By sending thus obtained image to a geographically separatedremote user using communication means such as the Internet, the user canselect his own viewpoint from the received omnidirectional imageaccording to his own personal interest, and image processing software onthe user's computing device can extract a partial image corresponding tothe user-selected viewpoint, and a perspectively correct planar imagecan be displayed on the computing device. Therefore, using the imageprocessing software, the user can make a choice of turning around (pan),looking-up or down (tilt), or taking a close (zoom in) or a remote (zoomout) view as if the user is actually present at the specific place inthe image. This method has a distinctive advantage of multiple usersaccessing the same Internet site being able to take looks along thedirections of their own choices. This advantage cannot be enjoyed in apanoramic imaging system employing a motion camera such as a pan•tiltcamera.

References 10 and 11 describe a method of obtaining an omnidirectionalimage providing the views of every direction centered on the observer.Despite the lengthy description of the invention, however, theprojection scheme provided by the said references is one kind ofequidistance projection schemes in essence. In other words, thetechniques described in the documents make it possible to obtain anomnidirectional image from a real environment or from a cubic panorama,but the obtained omnidirectional image follows an equidistanceprojection scheme only and its usefulness is thus limited.

On the other hand, reference 12 provides an algorithm for projecting anOmnimax movie on a semi-cylindrical screen using a fisheye lens.Especially, taking into account of the fact that the projection schemeof a fisheye lens mounted on a movie projector deviates from an idealequidistance projection scheme, a method is described for locating theposition of the object point on the film corresponding to a certainpoint on the screen whereon an image point is formed. Therefore, it ispossible to calculate what image has to be on the film in order toproject a particular image on the screen, and such an image on the filmis produced using a computer. Especially, since the lens distortion isalready reflected in the image-processing algorithm, a spectator nearthe movie projector can entertain himself with a satisfactory panoramicimage. Nevertheless, the real projection scheme of the fisheye lens inthe said reference is inconvenient to use because it has been modeledwith the real image height on the film plane as the independentvariable, and the zenith angle of the incident ray as the dependentvariable. Furthermore, unnecessarily, the real projection scheme of thefisheye lens has been modeled only with odd polynomials.

Reference 13 provides examples of stereo panoramic images produced byProfessor Paul Bourke. Each of the panoramic images follows acylindrical projection scheme, and a panoramic image of an imaginaryscene produced by a computer as well as a panoramic image produced by arotating slit camera are presented. For panoramic images produced by acomputer or produced by a traditional method of rotating slit camera,the lens distortion is not an important issue. However, rotating slitcamera cannot be used to take a real-time panoramic image (i.e., movie)of a real world.

-   [reference 1] J. F. Blinn and M. E. Newell, “Texture and reflection    in computer generated images”, Communications of the ACM, 19,    542-547 (1976).-   [reference 2] N. Greene, “Environment mapping and other applications    of world projections”, IEEE Computer Graphics and Applications, 6,    21-29 (1986).-   [reference 3] S. D. Zimmermann, “Omniview motionless camera    orientation system”, U.S. Pat. No. 5,185,667, date of patent Feb. 9,    1993.-   [reference 4] E. Gullichsen and S. Wyshynski, “Wide-angle image    dewarping method and apparatus”, U.S. Pat. No. 6,005,611, date of    patent Dec. 21, 1999.-   [reference 5] E. W. Weisstein, “Cylindrical Projection”,    http://mathworld.wolfram.com/CylindricalProjection.html.-   [reference 6] W. D. G. Cox, “An introduction to the theory of    perspective—part 1”, The British Journal of Photography, 4, 628-634    (1969).-   [reference 7] G. Kweon, K. Kim, Y. Choi, G. Kim, and S. Yang,    “Catadioptric panoramic lens with a rectilinear projection scheme”,    Journal of the Korean Physical Society, 48, 554-563 (2006).-   [reference 8] G. Kweon, Y. Choi, G. Kim, and S. Yang, “Extraction of    perspectively normal images from video sequences obtained using a    catadioptric panoramic lens with the rectilinear projection scheme”,    Technical Proceedings of the 10th World Multi-Conference on    Systemics, Cybernetics, and Informatics, 67-75 (Orlando, Fla., USA,    June, 2006).-   [reference 9] H. L. Martin and D. P. Kuban, “System for    omnidirectional image viewing at a remote location without the    transmission of control signals to select viewing parameters”, U.S.    Pat. No. 5,384,588, date of patent Jan. 24, 1995.-   [reference 10] F. Oxaal, “Method and apparatus for performing    perspective transformation on visible stimuli”, U.S. Pat. No.    5,684,937, date of patent Nov. 4, 1997.-   [reference 11] F. Oxaal, “Method for generating and interactively    viewing spherical image data”, U.S. Pat. No. 6,271,853, date of    patent Aug. 7, 2001.-   [reference 12] N. L. Max, “Computer graphics distortion for IMAX and    OMNIMAX projection”, Proc. NICOGRAPH, 137-159 (1983).-   [reference 13] P. D. Bourke, “Synthetic stereoscopic panoramic    images”, Lecture Notes in Computer Graphics (LNCS), Springer, 4270,    147-155 (2006).-   [reference 14] G. Kweon and M. Laikin, “Fisheye lens”, Korean patent    application 10-2008-0030184, date of filing Apr. 1, 2008.-   [reference 15] G. Kweon and M. Laikin, “Wide-angle lenses”, Korean    patent 10-0826571, date of patent Apr. 24, 2008.

DISCLOSURE Technical Problem

The purpose of the present invention is to provide image processingalgorithms for extracting natural looking panoramic images fromdigitized images acquired using a camera equipped with a wide-angle lensthat is rotationally symmetric about an optical axis and devicesimplementing such algorithms.

Technical Solution

The present invention provides image processing algorithms that areaccurate in principle based on geometrical optics principle regardingimage formation by wide-angle lenses with distortion and mathematicaldefinitions of panoramic images.

Advantageous Effects

Panoramic images, which appear most natural to the naked eye, can beobtained by accurately image processing the images obtained using arotationally symmetric wide-angle lens. Such panoramic imaging systemsand devices can be used not only in security•surveillance applicationsfor indoor and outdoor environments, but also in diverse areas such asvideo phone for apartment entrance door, rear view camera for vehicles,visual sensor for robots, and also it can be used to obtain panoramicphotographs using a digital camera.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a catadioptric panoramic imaging systemof a prior art.

FIG. 2 is a conceptual drawing of an exemplary raw panoramic imageacquired using the catadioptric panoramic imaging system schematicallyillustrated in FIG. 1.

FIG. 3 is an unwrapped panoramic image corresponding to the rawpanoramic image given in FIG. 2.

FIG. 4 is a conceptual drawing illustrating the shape of an object planeemployed in a rectilinear panoramic imaging system.

FIG. 5 is a conceptual drawing of a raw panoramic image corresponding tothe object plane in FIG. 4.

FIG. 6 is a schematic diagram illustrating the desirable location andthe size of panoramic image on an image sensor plane.

FIG. 7 is a conceptual drawing of the latitude and the longitude on acelestial sphere.

FIG. 8 is a conceptual drawing of a map with the equi-rectangularprojection scheme.

FIG. 9 is a conceptual drawing illustrating a cylindrical projectionscheme.

FIG. 10 is an exemplary unwrapped panoramic image following acylindrical projection scheme.

FIG. 11 is another exemplary unwrapped panoramic image following acylindrical projection scheme.

FIG. 12 is a conceptual drawing illustrating a projection scheme mostappropriate for the panoramic imaging system of the first embodiment ofthe present invention.

FIG. 13 is a conceptual drawing of an uncorrected image plane accordingto the first embodiment of the present invention.

FIG. 14 is a schematic diagram of a panoramic imaging system accordingto the first embodiment of the present invention.

FIG. 15 is a conceptual drawing of a processed image plane that isdisplayed on an image displays means according to the first embodimentof the present invention.

FIG. 16 is a conceptual drawing of a horizontal cross-section of anobject plane according to the first embodiment of the present invention.

FIG. 17 is a conceptual drawing of a vertical cross-section of an objectplane according to the first embodiment of the present invention.

FIG. 18 is a conceptual drawing illustrating a real projection scheme ofa general rotationally symmetric lens.

FIG. 19 is a diagram showing the optical structure of a fisheye lenswith an equidistance projection scheme along with the traces of rays.

FIG. 20 is a graph showing the real projection scheme of the fisheyelens in FIG. 19.

FIG. 21 is a graph showing the difference between the real projectionscheme of the fisheye lens in FIG. 19 and a fitted projection schemeusing the least square error method.

FIG. 22 is a conceptual drawing illustrating the conversion relationbetween the rectangular coordinate and the polar coordinate of an objectpoint on an uncorrected image plane.

FIG. 23 is a conceptual drawing of a digitized processed image plane.

FIG. 24 is a conceptual drawing of a digitized uncorrected image planefor understanding the principle of distortion correction.

FIG. 25 is a conceptual drawing for understanding the principle ofbilinear interpolation.

FIG. 26 is an exemplary fisheye image produced by a computer assumingthat a fisheye lens with an equidistance projection scheme has been usedto take the picture of an imaginary scene.

FIG. 27 is a panoramic image following a cylindrical projection schemeextracted from the fisheye image given in FIG. 26.

FIG. 28 is a schematic diagram showing the desirable size and thelocation of real image on an image sensor plane.

FIG. 29 is an exemplary image acquired using a fisheye lens.

FIG. 30 is a graph showing the real projection scheme of the fisheyelens used to acquire the fisheye image given in FIG. 29.

FIG. 31 is a panoramic image following a cylindrical projection schemeextracted from the fisheye image given in FIG. 29.

FIG. 32 is a panoramic image following an equi-rectangular projectionscheme extracted from the fisheye image given in FIG. 26.

FIG. 33 is a panoramic image following a Mercator projection schemeextracted from the fisheye image given in FIG. 26.

FIG. 34 is a conceptual drawing of a wide-angle camera of prior artsinstalled at an upper part of an interior wall for the purpose ofmonitoring the entire interior space.

FIG. 35 is a schematic diagram showing the desirable size and thelocation of real image on an image sensor plane, which is mostappropriate for the third embodiment of the present invention.

FIG. 36 is a conceptual drawing of an horizontal installation of anultra wide-angle camera for the purpose of monitoring the entireinterior space.

FIG. 37 is a schematic diagram of a car rear view camera employing apanoramic imaging system according to the fourth embodiment of thepresent invention.

FIG. 38 is an example of an imaginary fisheye image captured by aninclined imaging system.

FIG. 39 is an exemplary panoramic image extracted from the fisheye imagegiven in FIG. 38 according to the fourth embodiment of the presentinvention.

FIG. 40 is a schematic diagram of an imaging system for monitoring theentire 360° directions from a building or a large bus without any deadspot by employing panoramic imaging systems from the first thorough thefourth embodiments of the present invention.

FIG. 41 is a schematic diagram of a device according to the sixthembodiment of the present invention, wherein vertical lines in the worldcoordinate system are not vertical lines in the first rectangularcoordinate system.

FIG. 42 is an example of an imaginary image captured by an inclinedimaging system of the sixth embodiment of the present invention.

FIG. 43 is an exemplary panoramic image extracted from the image givenin FIG. 42 using the method described in the first embodiment of thepresent invention.

FIG. 44 is a schematic diagram showing the desirable size and thelocation of real image on an image sensor plane, which is mostappropriate for the sixth embodiment of the present invention.

FIG. 45 is a schematic diagram of an imaging system according to thesixth embodiment of the present invention.

FIG. 46 is a conceptual drawing illustrating the relation between thethird rectangular coordinate system and the world coordinate system inthe sixth embodiment of the present invention.

FIG. 47 is an exemplary panoramic image following a cylindricalprojection scheme according to the sixth embodiment of the presentinvention.

FIGS. 48 and 49 are conceptual drawings illustrating the relationbetween the first rectangular coordinate system and the world coordinatesystem in the seventh embodiment of the present invention.

FIG. 50 is an example of an imaginary image captured by an inclinedimaging system of the seventh embodiment of the present invention.

FIG. 51 is an exemplary panoramic image following a cylindricalprojection scheme according to the seventh embodiment of the presentinvention.

FIG. 52 is a conceptual drawing of an object plane according to theeighth embodiment of the present invention.

FIG. 53 is a diagram showing the optical structure of an exemplarydioptric fisheye lens with a stereographic projection scheme along withthe traces of rays.

FIG. 54 is a diagram showing the optical structure of an exemplarycatadioptric fisheye lens with a stereographic projection scheme alongwith the traces of rays.

FIG. 55 is a conceptual drawing of an incidence plane according to theeighth embodiment of the present invention.

FIG. 56 is a conceptual drawing of an uncorrected image plane accordingto the eighth embodiment of the present invention.

FIG. 57 is a conceptual drawing of a processed image plane according tothe eighth embodiment of the present invention.

FIGS. 58 and 59 are exemplary panoramic images according to the eighthembodiment of the present invention.

FIG. 60 is a conceptual drawing of an incidence plane according to theninth embodiment of the present invention.

FIGS. 61 and 62 are exemplary panoramic images according to the ninthembodiment of the present invention.

FIG. 63 is a conceptual drawing of the world coordinate system accordingto the tenth embodiment of the present invention.

FIG. 64 is a conceptual drawing of a processed image plane according tothe tenth embodiment of the present invention.

MODE FOR INVENTION

Hereinafter, referring to FIG. 12 through FIG. 64, the preferableembodiments of the present invention will be described in detail.

First Embodiment

FIG. 12 is a schematic diagram for understanding the field of view andthe projection scheme of a panoramic imaging system according to thefirst embodiment of the present invention. The panoramic imaging systemof the current embodiment is assumed as attached on a verticalwall(1230), which is perpendicular to the ground plane. The wallcoincides with the X-Y plane, and the Y-axis runs from the ground plane(i.e., X-Z plane) to the zenith. The origin of the coordinate is locatedat the nodal point N of the lens, and the optical axis(1201) of the lenscoincides with the Z-axis (in other words, it is parallel to the groundplane). Hereinafter, this coordinate system is referred to as the worldcoordinate system. The world coordinate system is a coordinate systemfor describing the environment around the camera that is captured by thelens, and it is a right-handed coordinate system.

In a rigorous sense, the direction of the optical axis is the directionof the negative Z-axis of the world coordinate system. This is because,by the notational convention of imaging optics, the direction from theobject (or, an object point) to the image plane (or, an image point) isthe positive direction. Despite this fact, we will describe the opticalaxis as coinciding with the Z-axis of the world coordinate system forthe sake of simplicity in argument. This is because the currentinvention is not an invention about a lens design but an invention usinga lens, and in the viewpoint of a lens user, it makes easier tounderstand by describing the optical axis as in the current embodimentof the present invention.

The image sensor plane(1213) is a plane having a rectangular shape andperpendicular to the optical axis, whereof the lateral dimension is B,and the longitudinal dimension is V. Here, we assume a first rectangularcoordinate system, wherein the nodal point N of the lens is taken as theorigin, and the optical axis(1201) is taken as the negative(−) z-axis.In other words, the direction of the z-axis is the exact oppositedirection of the Z-axis. The intersection point between the z-axis andthe image sensor plane(1213) is O. The x-axis of the first rectangularcoordinate system passes through the intersection point O and isparallel to the lateral side of the image sensor plane, and the y-axispasses through the intersection point O and is parallel to thelongitudinal side of the image sensor plane. Identical to the worldcoordinate system, this first rectangular coordinate system is aright-handed coordinate system.

In the current embodiment, the X-axis of the world coordinate system isparallel to the x-axis of the first rectangular coordinate system, andpoints in the same direction. On the other hand, the Y-axis of the worldcoordinate system is parallel to the y-axis of the first rectangularcoordinate system, but the direction of the Y-axis is the exact oppositeof the direction of the y-axis. Therefore, in FIG. 12, the positivedirection of the x-axis of the first rectangular coordinate system runsfrom the left to the right, and the positive direction of the y-axisruns from the top to the bottom.

The intersection point O between the z-axis of the first rectangularcoordinate system and the sensor plane(1213)—hereinafter referred to asthe first intersection point—is not generally located at the center ofthe sensor plane, and it can even be located outside the sensor plane.Such a case can happen when the center of the image sensor is moved awayfrom the center position of the lens—i.e., the optical axis—on purposein order to obtain an asymmetric vertical or horizontal field of view.

The lateral coordinate x of an arbitrary point P—hereinafter referred toas the first point—on the sensor plane(1213) has a minimum value x₁ anda maximum value x₂ (i.e., x₁≦x≦x₂). By definition, the differencebetween the maximum lateral coordinate and the minimum lateralcoordinate is the lateral dimension of the sensor plane (i.e., x₂−x₁=B).In the same manner, the longitudinal coordinate y of the first point Phas a minimum value y₁ and a maximum value y₂ (i.e., y₁≦y≦y₂). Bydefinition, the difference between the maximum longitudinal coordinateand the minimum longitudinal coordinate is the longitudinal dimension ofthe sensor plane (i.e., y₂−y₁=V).

However, it is not desirable to use a raw image acquired using a fisheyelens in order to obtain a horizontal field of view of 180°. This isbecause a natural-looking panoramic image cannot be obtained due to thepreviously mentioned barrel distortion. A panoramic lens assuming objectplanes schematically shown in FIG. 4 through FIG. 9 follows arectilinear projection scheme in the vertical direction, and by theinherent structure of the lens, follows an equidistance projectionscheme in the horizontal direction. Therefore, it is desirable that apanoramic imaging system of the current invention follows anequidistance projection scheme in the horizontal direction, and followsa rectilinear projection scheme in the vertical direction. Such aprojection scheme corresponds to assuming a hemi-cylindrical objectplane(1231) with a radius S and having the Y-axis as the rotationalsymmetry axis, and the image of an arbitrary point Q on the objectplane(1231)—hereinafter referred to as an object point—appears as animage point on the said sensor plane(1213). According to a desirableprojection scheme of the current invention, the image of an object onthe hemi-cylindrical object plane(1231) is captured on the sensorplane(1213) with its vertical proportions preserved, and the lateralcoordinate x of the image point is proportional to the horizontal arclength of the corresponding object point on the said object plane, andthe image points on the image sensor plane by all the object points onthe object plane(1231) collectively form a real image. When such acondition is satisfied, the image obtainable is, in effect, equivalentto selecting a portion of the image in FIG. 3 corresponding to ahorizontal FOV of 180°.

An arbitrary rotationally symmetric lens including a fisheye lens,however, does not follow the said projection scheme. Therefore, torealize the said projection scheme, an image processing stage isinevitable. FIG. 13 is a conceptual drawing of an uncorrected imageplane(1334) prior to the image processing stage, which corresponds tothe image sensor plane(1213). If the lateral dimension of the imagesensor plane(1213) is B and the longitudinal dimension is V, then thelateral dimension of the uncorrected image plane is gB and thelongitudinal dimension is gV, where g is proportionality constant.

Uncorrected image plane(1334) can be considered as the image displayedon the image display means without rectification of distortion, and is amagnified image of the real image on the image sensor plane by amagnification ratio g. For example, the image sensor plane of a ⅓-inchCCD sensor has a rectangular shape having a lateral dimension of 4.8 mm,and a longitudinal dimension of 3.6 mm. On the other hand, if the sizeof a monitor is 48 cm wide and 36 cm high, then the magnification ratiog is 100. More desirably, the side dimension of a pixel in a digitalimage is considered as 1. A VGA-grade ⅓-inch CCD sensor has pixels in atwo-dimensional array having 640 columns and 480 lows. Therefore, eachpixel has a right rectangular shape with both the width and the heightmeasuring as 4.8 mm/640=7.5 μm, and in this case, the magnificationratio g is given by 1 pixel/7.5 μm=133.3 pixel/mm. In recapitulation,the uncorrected image plane(1334) is a distorted digital image obtainedby converting the real image formed on the image sensor plane intoelectrical signals.

The said first intersection point O is the intersection point betweenthe optical axis(1201) and the image sensor plane(1213). Therefore, aray entered along the optical axis forms an image point on the saidfirst intersection point O. By definition, the horizontal incidenceangle ψ and the vertical incidence angle δ of a ray entered along theoptical axis are both zero. Therefore, the point O′ on the uncorrectedimage plane corresponding to the first intersection point O in the imagesensor plane—hereinafter referred to as the second intersectionpoint—corresponds to the image point by an incident ray having ahorizontal incidence angle of 0 as well as a vertical incidence angle of0.

A second rectangular coordinate systems is assumed wherein x′-axis istaken as the axis that passes through the said second intersection pointand is parallel to the lateral side of the uncorrected imageplane(1334), and y′-axis is taken as the axis that passes through thesaid second intersection point and is parallel to the longitudinal sideof the uncorrected image plane. In FIG. 13, the positive direction ofthe x′-axis runs from the left to the right, and the positive directionof the y′-axis runs from the top to the bottom. Then, the lateralcoordinate x′ of an arbitrary point P′ on the uncorrected imageplane(1334) has a minimum value x′₁=gx₁ and a maximum valuex′₂=gx₂(i.e., gx₁≦x′≦gx₂). In the same manner, the longitudinalcoordinate y′ of the said point has a minimum value y′₁=gy₁ and amaximum value y′₂=gy₂(i.e., gy₁≦y′≦gy₂).

As has been described, a fisheye lens does not provide a natural-lookingpanoramic image as is schematically shown in FIG. 12. The main point ofthe current embodiment is about a method of obtaining a mostnatural-looking panoramic image as is schematically shown in FIG. 12 byapplying a mathematically accurate image processing algorithm on adistorted image obtained using a rotationally symmetric wide-angle lens,which includes a fisheye lens. FIG. 14 is a schematic diagram of adevice using the image processing methods of the current embodiments ofthe present invention, wherein the device has an imaging system whichmainly includes an image acquisition means(1410), an image processingmeans(1416) and image display means(1415, 1417). The image acquisitionmeans(1410) includes a rotationally symmetric wide-angle lens(1412) anda camera body(1414) having an image sensor(1413) inside. The saidwide-angle lens can be a fisheye lens with more than 180° FOV and havingan equidistance projection scheme, but it is by no means limited to sucha fisheye lens. Hereinafter, for the sake of simplicity in argument, awide-angle lens is referred to as a fisheye lens. Said camera body iseither a digital camera or a video camera that can produce movie files,and contains an image sensor such as CCD or CMOS sensor. In thisembodiment, the optical axis(1401) of the said fisheye lens is parallelto the ground plane(1417). In other embodiments, the optical axis can beperpendicular to the ground plane, or it can be inclined at an arbitraryangle. By the said fisheye lens(1412), a real image(1433: marked as asolid line in the figure) of the object plane(1431) is formed on thefocal plane(1432: marked as a dotted line). In order to obtain a sharpimage, the image sensor plane(1413) must coincide with the focalplane(1432).

The real image(1433) of the objects on the object plane(1431) formed bythe fisheye lens(1412) is converted by the image sensor(1413) intoelectrical signals, and displayed as an uncorrected image plane(1434) onthe image display means(1415), wherein this uncorrected image planecontains a distortion aberration. If the said lens is a fisheye lens,then the distortion will be mainly a barrel distortion. This distortedimage can be rectified by the image processing means(1416), and thendisplayed as a processed image plane(1435) on an image displaymeans(1417) such as a computer monitor or a CCTV monitor. Said imageprocessing can be a software image processing by a computer, or ahardware image processing by an FPGA (Field Programmable Gate Array).The following table 1 summarizes corresponding variables in the objectplane, the image sensor plane, the uncorrected image plane, and theprocessed image plane.

TABLE 1 image sensor uncorrected processed image surface object planeplane image plane plane lateral dimension of the L B gB W planelongitudinal dimension T V gV H of the plane coordinate system world thefirst the second the third rectangular coordinate rectangularrectangular coordinate system system coordinate coordinate location ofthe nodal point of nodal point of the nodal point of nodal point of thecoordinate origin the lens lens the lens lens symbol of the O O′ O″intersection point with the optical axis coordinate axes (X, Y, Z) (x,y, z) (x′, y′, z′) (x″, y″, z″) name of the object object point thefirst point the second point the third point point or the image pointssymbol of the object Q P P′ P″ point or the image point two-dimensional(x, y) (x′, y′) (x″, y″) coordinate of the object point or the imagepoint

FIG. 15 is a conceptual drawing of a rectified screen of the currentinvention, wherein the distortion has been removed. In other words, itis a conceptual drawing of a processed image plane(1535). The processedimage plane(1535) has a rectangular shape, whereof the lateral sidemeasures as W and the longitudinal side measures as H. Furthermore, athird rectangular coordinate system is assumed wherein x″-axis isparallel to the lateral side of the processed image plane, and y″-axisis parallel to the longitudinal side of the processed image plane. Thez″-axis of the third rectangular coordinate system is parallel to thez-axis of the first rectangular coordinate system and the z′-axis of thesecond rectangular coordinate system. The intersection point O″ betweenthe said z″-axis and the processed image plane can take an arbitraryposition, and it can even be located outside the processed image plane.In FIG. 15, the positive direction of the x″-axis runs from the left tothe right, and the positive direction of the y″-axis runs from the topto the bottom. Here, the lateral coordinate x″ of a third point P″ onthe processed image plane(1535) has a minimum value x″, and a maximumvalue x″₂(i.e., x″₁≦x″≦x″₂). By definition, the difference between themaximum lateral coordinate and the minimum lateral coordinate is thelateral dimension of the processed image plane (i.e., x″₂−x″₁=W). In thesame manner, the longitudinal coordinate y″ of the third point P″ has aminimum value y″₁ and a maximum value y″₂ (i.e., y″₁≦y″≦y″₂). Bydefinition, the difference between the maximum longitudinal coordinateand the minimum longitudinal coordinate is the longitudinal dimension ofthe processed image plane (i.e., y″₂−y″₁=H).

FIG. 16 shows the cross-section of the object plane in FIG. 12 in theX-Z plane. The horizontal FOV of the imaging system of the presentinvention is not necessarily 180°, and it can be smaller or larger thanthat. In this reason, illustrated in FIG. 16 is a case where the FOV islarger than 180°. The horizontal angle of incidence of an arbitraryincident ray(1605) impinging on the imaging system of the presentinvention, which is the angle subtended by the incident ray and the Y-Zplane, is ψ. In other words, it is the incidence angle in the lateraldirection with respect to the Z-axis (i.e., the optical axis) in the X-Zplane (i.e., the ground plane). Conventionally, an incident ray forminga sharp image on the focal plane by the imaging properties of a lens isassumed as to pass through the nodal point N of the lens.

The minimum value of the horizontal incidence angle is ψ₁, the maximumincidence angle is ψ₂ (i.e., ψ₁≦ψ≦ψ₂), and the horizontal FOV isΔψ=ψ₂−ψ₁. In general, if the horizontal FOV is 180°, then a desirablerange of the horizontal incidence angle will be given by ψ₂=−ψ₁=90°.Since the radius of the object plane is S, the arc length of the saidobject plane is given by Eq. 17.

L=S(ψ₂−ψ₁)=SΔψ  [Math Figure 17]

Here, it has been assumed that the unit of the field of view Δψ isradian. This arc length L must be proportional to the lateral dimensionW of the processed image plane. Therefore, if this proportionalityconstant is c, then the following equation 18 is satisfied.

L=cW  [Math Figure 18]

On the other hand, FIG. 17 shows the cross-section of the object planein FIG. 12 in the Y-Z plane. The radius of the said object plane(1731)is S, and the height of the object plane is T. The vertical incidenceangle of an incident ray(1705) entering into a lens of the presentinvention, which is the angle with respect to the Z-axis (i.e., theoptical axis) in the Y-Z plane (i.e., a plane containing a verticalline), is δ. In other words, the vertical incidence angle the saidincident ray(1705) makes with the X-Z plane is δ. The minimum value ofthis vertical incidence angle is δ₁, and the maximum value is δ₂ (i.e.,δ₁≦δ≦δ₂). When the vertical FOV is Δδ=δ₂−δ₁, it is simpler if the rangeof the vertical incidence angle is given as δ₂=−δ₁=Δδ/2, but accordingto the needs, it may be desirable if the two values are different. Forexample, if it is installed on the roof of a vehicle, then it isdesirable to mainly monitor the area above the horizon, but if it isinstalled on an airplane, it is desirable to mainly monitor the areabelow the horizon. Here, the height T of the object plane seen from theorigin N of the said coordinate system is given by Eq. 19.

T=S(tan δ₂−tan δ₁)  [Math Figure 19]

Furthermore, the height T of the object plane must satisfy the sameproportionality relation with the height H of the processed image plane.

T=cH  [Math Figure 20]

Equation 21 can be obtained from Eqs. 17 and 18, wherein A is aconstant.

$\begin{matrix}{{A \equiv \frac{S}{c}} = \frac{W}{\Delta \; \psi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 21} \right\rbrack\end{matrix}$

On the other hand, Eq. 22 can be obtained from Eqs. 19 and 20.

$\begin{matrix}{A = \frac{H}{{\tan \; \delta_{2}} - {\tan \; \delta_{1}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 22} \right\rbrack\end{matrix}$

Therefore, from Eqs. 21 and 22, it can be seen that the followingequation must be satisfied.

$\begin{matrix}{\frac{W}{H} = \frac{\Delta \; \psi}{{\tan \; \delta_{2}} - {\tan \; \delta_{1}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 23} \right\rbrack\end{matrix}$

In most of the cases, it will be desirable if the range of thehorizontal incidence angle and the range of the vertical incidence angleare symmetrical. Therefore, the horizontal FOV will be given asΔψ=ψ₂−ψ₁=2ψ₂, and the vertical FOV will be given as Δδ=δ₂−δ₁=2 δ₂. Whendesigning a lens or evaluating the characteristics of a lens, thehorizontal FOV Δψ and the vertical FOV Δδ are important barometers. FromEq. 23, it can be seen that the vertical FOV must be given as in Eq. 24as a function of the horizontal FOV.

$\begin{matrix}{{\Delta \; \delta} = {2\; {\tan^{- 1}\left( {\frac{H}{2\; W}\Delta \; \psi} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 24} \right\rbrack\end{matrix}$

For example, if we assume that the horizontal FOV of the imaging systemis 180°, and an ordinary image sensor plane having the 4:3 aspect ratiobetween the lateral dimension and the longitudinal dimension isemployed, then the vertical FOV of a natural panoramic image is given byEq. 25.

$\begin{matrix}{{\Delta \; \delta} = {{2{\tan^{- 1}\left( {\frac{3}{8}\pi} \right)}} = {99.35{^\circ}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 25} \right\rbrack\end{matrix}$

On the other hand, if we assume an image sensor having the 16:9 ratio,then the vertical FOV is given by Eq. 26.

$\begin{matrix}{{\Delta \; \delta} = {{2\; {\tan^{- 1}\left( {\frac{9}{32}\pi} \right)}} = {82.93{^\circ}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 26} \right\rbrack\end{matrix}$

Therefore, even if an image sensor having the 16:9 ratio is used, thevertical FOV corresponds to an ultra wide-angle.

More generally, when the procedure from Eq. 17 through Eq. 23 isrepeated on an interval containing the third intersection point O″, thenEq. 27 can be obtained.

$\begin{matrix}\begin{matrix}{A = {\frac{H}{{\tan \; \delta_{2}} - {\tan \; \delta_{1}}} = {\frac{y^{''}}{\tan \; \delta} = {\frac{y_{2}^{''}}{\tan \; \delta_{2}} = \frac{y_{1}^{''}}{\tan \; \delta_{1}}}}}} \\{= {\frac{W}{\Delta \; \psi} = {\frac{x^{''}}{\psi} = {\frac{x_{1}^{''}}{\psi_{1}} = \frac{x_{2}^{''}}{\psi_{2}}}}}}\end{matrix} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 27} \right\rbrack\end{matrix}$

Therefore, when setting-up the desirable size of the processed imageplane and the FOV, it must be ensured that Eq. 27 is satisfied.

If the processed image plane in FIG. 15 satisfies the said projectionscheme, then the horizontal incidence angle of an incident raycorresponding to the lateral coordinate x″ of a third point P″ on thesaid processed image plane is given by Eq. 28.

$\begin{matrix}{\psi = {{\frac{\Delta \; \psi}{W}x^{''}} = \frac{x^{''}}{A}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 28} \right\rbrack\end{matrix}$

Likewise, the vertical incidence angle of an incident ray correspondingto the third point having a longitudinal coordinate y″ is, from Eq. 27,given as Eq. 29.

$\begin{matrix}{\delta = {\tan^{- 1}\left( \frac{y^{''}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 29} \right\rbrack\end{matrix}$

Therefore, the signal value of a third point on the processed imageplane having an ideal projection scheme must be given as the signalvalue of an image point on the image sensor plane formed by an incidentray originating from an object point on the object plane having ahorizontal incidence angle (i.e., the longitude) given by Eq. 28 and avertical incidence angle (i.e., the latitude) given by Eq. 29.

The location of the object point Q on the object plane having saidhorizontal and vertical incidence angles can be obtained by thefollowing method. Referring to FIG. 7, a vector from the origin N of theworld coordinate system to an object point Q on the object plane havingsaid horizontal and vertical incidence angles can be written as R. Thedirection of this vector is the exact opposite of the propagationdirection of the incident ray(1605, 1705), and the said vector in theworld coordinate system can be written as Eq. 30.

R=X{circumflex over (X)}+YŶ+Z{circumflex over (Z)}  [Math Figure 30]

In Eq. 30, {circumflex over (X)}=(1,0,0) is the unit vector along theX-axis, and likewise, Ŷ=(0,1,0) and {circumflex over (Z)}=(0,0,1) arethe unit vectors along the Y-axis and the Z-axis, respectively. On theother hand, the said vector R can be given in the spherical polarcoordinate system as a function of the zenith angle θ and the azimuthangle φ as given in Eq. 31.

R=R{circumflex over (R)}(θ,φ)  [Math Figure 31]

Here, R is the magnitude of the said vector R, and {circumflex over (R)}is the direction vector of the said vector. Then, the following relationholds between the rectangular coordinate and the spherical polarcoordinate.

X={circumflex over (X)}· R=R sin θ cos φ  [Math Figure 32]

Y=Ŷ· R=R sin θ sin φ  [Math Figure 33]

Z={circumflex over (Z)}· R=R cos θ  [Math Figure 34]

R· R=X ² +Y ² +Z ² =R ²  [Math Figure 35]

In Eqs. 32 through 35, dot(·) represent a scalar product.

On the other hand, the said direction vector can be given by Eq. 36 as afunction of two incidence angles describing the projection scheme of thecurrent invention, namely the horizontal incidence angle ψ and thevertical incidence angle δ. Hereinafter, this coordinate system will bereferred to as a cylindrical polar coordinate system.

R=R{circumflex over (R)}(ψ,δ)  [Math Figure 36]

Using these two incidence angles, the rectangular coordinate can begiven as follows.

X=R cos δ sin ψ  [Math Figure 37]

Y=R sin δ  [Math Figure 38]

Z=R cos δ cos ψ  [Math Figure 39]

Using Eqs. 37 through 39, the horizontal and the vertical incidenceangles can be obtained from the rectangular coordinate (X, Y, Z) of theobject point as in Eqs. 40 and 41.

$\begin{matrix}{\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 40} \right\rbrack \\{\delta = {\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}\;} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 41} \right\rbrack\end{matrix}$

On the other hand, since the coordinates given in the spherical polarcoordinate system and in the cylindrical polar coordinate system mustagree, the following relations given in Eqs. 42 through 44 must hold.

sin θ cos θ=cos δ sin ψ  [Math Figure 42]

sin θ sin φ=sin δ  [Math Figure 43]

cos θ=cos δ cos ψ  [Math Figure 44]

Eq. 45 can be obtained by dividing Eq. 43 by Eq. 42.

$\begin{matrix}{{\tan \; \varphi} = \frac{\tan \; \delta}{\sin \; \psi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 45} \right\rbrack\end{matrix}$

Therefore, the azimuth angle φ is given by Eq. 46.

$\begin{matrix}{\varphi = {\tan^{- 1}\left( \frac{\tan \; \delta}{\sin \; \psi} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 46} \right\rbrack\end{matrix}$

On the other hand, from Eq. 44, the zenith angle θ is given by Eq. 47.

θ=cos⁻¹(cos δ cos ψ)  [Math Figure 47]

In the reverse direction, to convert from the spherical polar coordinateto the cylindrical polar coordinate, Eq. 48 can be obtained by dividingEq. 42 by Eq. 44.

tan ψ=tan θ cos φ  [Math Figure 48]

Therefore, the horizontal incidence angle is given by Eq. 49.

ψ=tan⁻¹(tan θ cos φ)  [Math Figure 49]

On the other hand, from Eq. 43, the vertical incidence angle is given byEq. 50.

δ=sin⁻¹(sin θ sin φ)  [Math Figure 50]

Therefore, an incident ray having a horizontal incidence angle ψ and avertical incidence angle δ is an incident ray in the spherical polarcoordinate system having a zenith angle θ given by Eq. 47 and an azimuthangle φ given by Eq. 46. In order to process an image, the position onthe image sensor plane corresponding to an incident ray having such azenith angle θ and an azimuth angle φ must be determined.

FIG. 18 is a conceptual drawing illustrating the real projection schemesof rotationally symmetric wide-angle lenses(1812) including fisheyelenses. The optical axis(1801) of the lens(1812) of the presentembodiment coincides with the Z-axis of the coordinate system. Anincident ray(1805) having a zenith angle θ with respect to the Z-axis isrefracted by the lens(1812) and forms an image point P—i.e., the firstpoint—on the sensor plane(1813) within the camera body(1814). The saidimage sensor plane(1813) is perpendicular to the optical axis, and toobtain a sharp image, the sensor plane must coincide with the focalplane(1832) of the lens. The distance between the said image point P andthe intersection point O between the optical axis(1801) and the sensorplane(1813), in other words, the first intersection point, is r.

For a fisheye lens with an ideal equidistance projection scheme, theimage height r is given by Eq. 51.

r(θ)=fθ  [Math Figure 51]

In Eq. 51, the unit of the incidence angle θ is radian, and f is theeffective focal length of the fisheye lens. The unit of the image heightr is identical to the unit of the effective focal length f. Thedifference between the ideal equidistance projection scheme given by Eq.51 and the real projection scheme of the lens is the f−θ distortion.Nevertheless, it is considerably difficult for a fisheye lens tofaithfully implement the projection scheme given by Eq. 51, and thediscrepancy can be as large as 10%. Furthermore, the applicability ofthe present image processing algorithm is not limited to a fisheye lenswith an equidistance projection scheme. Therefore, it is assumed thatthe projection scheme of a lens is given as a general function of thezenith angle θ of the incident ray as given in Eq. 52.

r=r(θ)  [Math Figure 52]

This function is a monotonically increasing function of the zenith angleθ of the incident ray.

Such a real projection scheme of a lens can be experimentally measuredusing an actual lens, or can be calculated from the lens prescriptionusing dedicated lens design software such as Code V or Zemax. Forexample, the y-axis coordinate y of an image point on the focal plane byan incident ray having given horizontal and vertical incidence anglescan be calculated using a Zemax operator REAY, and the x-axis coordinatex can be similarly calculated using an operator REAX.

FIG. 19 is a diagram showing the optical structure of a fisheye lenswith an equidistance projection scheme along with the traces of rays, ofwhich the complete lens prescription is given in reference 14. The FOVof this fisheye lens is 190°, the F-number is 2.8, and it has an enoughresolution for a VGA-grade camera simultaneously in the visible and thenear infrared wavelengths. Furthermore, the relative illumination isquite fair being over 0.8. This lens is comprised of only 8 sphericallens elements, and is appropriate for mass production since it has anenough fabrication tolerance.

FIG. 20 shows the real projection scheme (dotted line) of the fisheyelens given in FIG. 19 in the visible wavelength range, along with thefitted projection scheme (solid line) by a polynomial function. Here,the real projection scheme has been obtained using the said REAYoperator based on the complete lens prescription, and this realprojection scheme has been fitted to a fifth order polynomial passingthrough the origin as given in Eq. 53.

r(θ)=α₁θ+α₂θ²+α₃θ³+α₄θ⁴+α₅θ⁵  [Math Figure 53]

Table 2 shows the polynomial coefficients in Eq. 53.

TABLE 2 variable value a₁ 1.560778 a₂   3.054932 × 10⁻² a₃ −1.078742 ×10⁻¹ a₄   7.612269 × 10⁻² a₅ −3.101406 × 10⁻²

FIG. 21 shows the error between the real projection scheme and theapproximate projection scheme fitted to a polynomial given by Eq. 53 andtable 2. As can be seen from FIG. 21, the error is less than 0.3 μm, andit is practically error-free considering the fact that the dimension ofeach side of a pixel is 7.5 μm in a VGA-grade ⅓-inch CCD sensor.

FIG. 22 is a conceptual drawing illustrating the conversion relationbetween the rectangular coordinate and the polar coordinate of thesecond point P′ on the uncorrected image plane(2234) corresponding tothe first point on the sensor plane. Referring to FIG. 22, thetwo-dimensional rectangular coordinate (x′, y′) of the second point onthe uncorrected image plane can be obtained from the two-dimensionalpolar coordinate (r′, φ′≡φ) as in Eqs. 54 and 55.

x′=gr(θ)cos φ  [Math Figure 54]

y′=gr(θ)sin φ  [Math Figure 55]

Using Eqs. 27 through 55, a panoramic image having an ideal projectionscheme can be extracted from an image acquired using a fisheye lensexhibiting a distortion aberration. First, depending on the user's need,a desirable size (W, H) of the panoramic image and the location of thethird intersection point O″ are determined. The said third intersectionpoint can be located even outside the processed image plane. In otherwords, the range of the lateral coordinate (x″₁≦x″≦x″₂) on the processedimage plane as well as the range of the longitudinal coordinatey″₁≦y″≦y″₂) can take arbitrary real numbers. Also, the horizontal FOV Δψof this panoramic image (i.e., the processed image plane) is determined.Then, the horizontal incidence angle ψ and the vertical incidence angleδ of an incident ray corresponding to the third point in the panoramicimage having a rectangular coordinate (x″, y″) can be obtained usingEqs. 28 and 29. Then, the zenith angle θ and the azimuth angle φ of anincident ray having the said horizontal and the vertical incidenceangles are calculated using Eqs. 47 and 46. Next, the real image heightr corresponding to the zenith angle θ of the incident ray is obtainedusing Eq. 52. Utilizing the real image height r, the magnification ratiog, and the azimuth angle φ of the incident ray, the rectangularcoordinate (x′, y′) of the image point on the uncorrected image plane isobtained using Eqs. 54 and 55. In this procedure, the coordinate of thesecond intersection point on the uncorrected image plane, orequivalently the location of the first intersection point on the sensorplane has to be accurately determined. Such a location of theintersection point can be easily found using various methods includingimage processing method. Since such technique is well known to thepeople in this field, it will not be described in this document.Finally, the video signal (i.e., RGB signal) from the image point by thefisheye lens having this rectangular coordinate is given as the videosignal for the image point on the panoramic image having the rectangularcoordinate (x″, y″). A panoramic image having an ideal projection schemecan be obtained by image processing for all the image points on theprocessed image plane by the above-described method.

However, in reality, a complication arises due to the fact that all theimage sensors and display devices are digitized devices. FIG. 23 is aschematic diagram of a digitized processed image plane, and FIG. 24 is aschematic diagram of an uncorrected image plane. Said processed imageplane has pixels in the form of a two-dimensional array having J_(max)columns in the lateral direction and I_(max) lows in the longitudinaldirection. Although, in general, each pixel has a square shape with boththe lateral dimension and the longitudinal dimension measuring as p, thelateral and the longitudinal dimensions of a pixel are considered as 1in the image processing field. To designate a particular pixel P″, thelow number I and the column number J are used. In FIG. 23, thecoordinate of the said pixel P″ is given as (I, J). Therefore, thesignal stored on this pixel can be designated as S(I, J). A pixel has afinite area. To correct the distortion of a digitized image, thephysical coordinate of an arbitrary pixel P″ is taken as the centerposition of the pixel, which is marked as a filled circle in FIG. 23.

There is an image point—i.e., the first point—on the image sensor planecorresponding to a pixel P″ on the said processed image plane(2335). Thehorizontal incidence angle of an incident ray in the world coordinatesystem forming an image at this first point can be written asψ_(I,J)≡ψ(I, J). Also, the vertical incidence angle can be written asδ_(I,J)≡δ(I, J). Incidentally, the location of this first point does notgenerally coincide with the exact location of any one pixel.

Here, if the said screen(2335) corresponds to a panoramic image, then asgiven by Eq. 56, the horizontal incidence angle must be a sole functionof the lateral pixel coordinate J.

ψ_(I,J)=ψ_(J)≡ψ(J)  [Math Figure 56]

Likewise, the vertical incidence angle must be a sole function of thelongitudinal pixel coordinate I.

δ_(I,J)=δ_(I)≡δ(I)  [Math Figure 57]

Furthermore, if an equidistance projection scheme is satisfied in thelateral direction, and a rectilinear projection scheme is satisfied inthe longitudinal direction, then the range of the horizontal incidenceangle and the range of the vertical incidence angle must satisfy therelation given in Eq. 58.

$\begin{matrix}{\frac{J_{{ma}\; x} - 1}{\psi_{J\; {ma}\; x} - \psi_{1}} = \frac{I_{{ma}\; x} - 1}{{\tan \; \delta_{I\; {ma}\; x}} - {\tan \; \delta_{1}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 58} \right\rbrack\end{matrix}$

Comparing with the image correction method described previously, imagecorrection method for a digitized image goes through the followingprocedure. First, the real projection scheme of the wide-angle lens thatis meant to be used in the image processing is obtained either byexperiment or based on the accurate lens prescription. Herein, when anincident ray having a zenith angle θ with respect to the optical axisforms a sharp image point on the image sensor plane by the image formingproperties of the lens, the real projection scheme of the lens refers tothe distance r from the intersection point O between the said imagesensor plane and the optical axis to the said image point obtained as afunction of the zenith angle θ of the incident ray.

r=r(θ)  [Math Figure 59]

Said function is a monotonically increasing function of the zenith angleθ. Next, the location of the optical axis on the uncorrected imageplane, in other words, the location of the second intersection point O′corresponding to the first intersection point O on the image sensorplane is obtained. The pixel coordinate of this second intersectionpoint is assumed as (K_(o), L_(o)). In addition to this, themagnification ratio g of the pixel distance r′ on the uncorrected imageplane over the real image height r on the image sensor plane isobtained. This magnification ratio g is given by Eq. 60.

$\begin{matrix}{g = \frac{r^{\prime}}{r}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 60} \right\rbrack\end{matrix}$

Once such a series of preparatory stages have been completed, then acamera mounted with the said fisheye lens is installed with its opticalaxis aligned parallel to the ground plane, and a raw image (i.e., anuncorrected image plane) is acquired. Next, the desirable size of theprocessed image plane and the location (I_(o), J_(o)) of the thirdintersection point is determined, and then the horizontal incidenceangle ψ_(J) and the vertical incidence angle δ_(I) given by Eqs. 61 and62 are computed for all the pixels (I, J) on the said processed imageplane.

$\begin{matrix}{\psi_{J} = {\frac{\psi_{J\; {ma}\; x} - \psi_{1}}{J_{{ma}\; x} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 61} \right\rbrack \\{\delta_{I} = {\tan^{- 1}\left\{ {\frac{\psi_{J\; {ma}\; x} - \psi_{1}}{J_{m\; {ax}} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 62} \right\rbrack\end{matrix}$

From these horizontal and vertical incidence angles, the zenith angleθ_(I,J) and the azimuth angle Φ_(I,J) of the incident ray in the firstrectangular coordinate system are obtained using Eqs. 63 and 64.

$\begin{matrix}{\theta_{I,J} = {\cos^{- 1}\left( {\cos \; \delta_{I\;}\cos \; \psi_{J}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 63} \right\rbrack \\{\varphi_{I,J} = {\tan^{- 1}\left( \frac{\tan \; \delta_{I}}{\sin \; \psi_{J}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 64} \right\rbrack\end{matrix}$

Next, the image height r_(I,J) on the image sensor plane is obtainedusing Eqs. 63 and 59.

r _(I,J) =r(θ_(I,J))  [Math Figure 65]

Next, using the location (K_(o), L_(o)) of the second intersection pointon the uncorrected image plane and the magnification ratio g, thelocation of the second point(2407) on the uncorrected image plane isobtained using Eqs. 66 and 67.

x′ _(I,J) =L _(o) +gr _(I,J) cos φ_(I,J)  [Math Figure 66]

y′ _(I,J) =K _(o) +gr _(I,J) sin φ_(I,J)  [Math Figure 67]

The location of the said second point does not exactly coincide with thelocation of any one pixel. Therefore, (x′_(I,J), y′_(I,J)) can beconsidered as the coordinate of a virtual pixel on the uncorrected imageplane corresponding to the third point on the processed image plane, andhas a real number in general.

Since the said second point does not coincide with any one pixel, aninterpolation method must be used for image processing. The coordinateof a pixel(2408) that is nearest to the location of the said secondpoint can be obtained using Eqs. 68 and 69.

K=int(y′ _(I,J))  [Math Figure 68]

L=int(x′ _(I,J))  [Math Figure 69]

Here, int{x} is a function whose output is an integer closest to thereal number x. Then, the signal value P(K, L) stored at that pixel iscopied and assigned as the signal value S(I, J) for the correspondingpixel in the unwrapped panoramic image.

S(I,J)=P(K,L)  [Math Figure 70]

Such a geometrical transformation is very simple, but it has theadvantage of fast execution time even for a case where there is a largenumber of pixels in the panoramic image.

Unwrapped panoramic image that is image processed by this most simplemethod has a shortcoming in that the boundary between two differentobjects appear jagged like that of a saw tooth when the number of pixelsis not sufficient in the image sensor, or when the unwrapped panoramicimage is magnified. To remedy this shortcoming, bilinear interpolationmethod can be employed. Referring to FIG. 25, the position (x′, y′) ofthe second point(2507) on the distorted uncorrected image planecorresponding to the pixel P″ having a coordinate (I, J) on thedistortion-free processed image plane is marked as a dark triangle.Here, the lateral and the longitudinal intervals between neighboringpixels are all 1, and the said second point is separated from thepixel(2508) by Δ_(x) pixel in the lateral direction and by Δ_(y) pixelin the longitudinal direction. Said pixel(2508) is a pixel, whereof thecoordinate is given by an integer value (K, L) that is obtained bytruncating the real numbered coordinate of the said second point. Inother words, if x′ is given as x′=103.9, for example, then int(x′)=104,but this value is not used in the bilinear interpolation method.Instead, the largest integer 103 which is smaller than x′ becomes the Lvalue. In mathematical terminology, it is designated as L=floor(x′) andK=floor(y′). Then, the signal value of the pixel calculated using thebilinear interpolation method is given by Eq. 71.

S(I,J)=(1−Δ_(y))(1−Δ_(x))P(K,L)+Δ_(y)(1−Δ_(x))P(K+1,L)+(1−Δ_(y))Δ_(x)P(K,L+1)+Δ_(y)Δ_(x) P(K+1,L+1)  [Math Figure 71]

When such a bilinear interpolation method is used, the image becomessharper. However, since the computational load is increased, it canbecome an obstacle in an imaging system operating in real time such asthe one generating live video signals. On the other hand, ifinterpolation methods such as bi-cubic interpolation or splineinterpolation methods are used, then a more satisfactory image can beobtained, but the computational load is increased still more. To preventthe decrease in speed (i.e., frame rate) due to such an increase in thecomputational load, the image processing can be done by hardware such asFPGA chips.

On the other hand, still other problems may occur when the image ismagnified or downsized, in other words, when the image is zoomed-in orzoomed-out by software. For example, the image can appear blurry whenthe image is excessively scaled up, and moire effect can happen when theimage is scaled down. Furthermore, due to an image processing, the imagecan be scaled-up and scaled-down within the same screen and the twoadverse effects can simultaneously appear. To improve on these problems,filtering operations can be undertaken in general.

FIG. 26 is an imaginary interior scene produced by professor Paul Bourkeby using a computer, and it has been assumed that the lens used tocapture the imaginary scene is a fisheye lens with 180°FOV having anideal equidistance projection scheme. This image is a square image,whereof both the lateral and the longitudinal dimensions are 250 pixels.Therefore, the coordinate of the optical axis, in other words, thecoordinate of the second intersection point, is (125.5, 125.5), and theimage height for an incident ray with a zenith angle of 90° is given asr′(π/2)=125.5−1=124.5. Since this imaginary fisheye lens follows anequidistance projection scheme, the projection scheme of this lens isgiven by Eq. 72.

$\begin{matrix}{{r^{\prime}(\theta)} = {{\frac{124.5}{\left( \frac{\pi}{2}\; \right)}\theta} = {79.26\theta}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 72} \right\rbrack\end{matrix}$

On the other hand, FIG. 27 is a panoramic image following a cylindricalprojection scheme that has been extracted from the image in FIG. 26,where the lateral and the longitudinal dimensions are all 250 pixels,and the third intersection point is located at the center of theprocessed image plane. Furthermore, the horizontal FOV of the processedimage plane is 180° (i.e., π). As can be seen from FIG. 27, all thevertical lines in the three walls, namely the front, the left, and theright walls in FIG. 26 appear as straight lines in FIG. 27. The factthat all the vertical lines in the world coordinate system appear asstraight lines in the processed image plane is the characteristic of thepresent invention.

FIG. 28 is a diagram schematically illustrating the desirable size andthe location of real image(2833) on the image sensor plane(2813). Thefisheye lens suitable for the present embodiment is a fisheye lens witha FOV greater than 180° and following an equidistance projection scheme.Furthermore, the intersection point O between the optical axis of thefisheye lens and the image sensor plane is located at the center of theimage sensor plane. Therefore, the range of the lateral coordinate is(−B/2≦x≦B/2), and the range of the longitudinal coordinate is(−V/2≦y≦V/2).

If the maximum FOV of this fisheye lens is given as 2θ₂, then the imageheight of an incident ray at the image sensor plane having the maximumzenith angle is given as r₂≡r(θ₂). Here, the desirable image height isgiven by Eq. 73.

$\begin{matrix}{{r\left( \theta_{2} \right)} = \frac{B}{2}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 73} \right\rbrack\end{matrix}$

Therefore, the image circle(2833) contacts the left edge(2813L) and theright edge(2813R) of the image sensor plane(2813). In thisconfiguration, the imaging system uses the most out of the pixels on theimage sensor plane and provides a satisfactory processed image plane.

FIG. 29 is an exemplary fisheye image acquired using a commercialfisheye lens having a 185°FOV, and FIG. 30 is an experimentally measuredreal projection scheme of this fisheye lens, wherein the image height onthe unprocessed image plane is given as a function of the zenith angle θof the incident ray as r′(θ)=gr(θ). If this fisheye lens is an idealfisheye lens, then the graph in FIG. 30 must be given as a straightline. However, it can be noticed that the graph deviates considerablyfrom a straight line. The real projection scheme shown in FIG. 30 can begiven as a simple polynomial of the zenith angle θ as given in Eq. 74.

r(θ)=α₁θ+α₂θ²+α₃θ³  [Math Figure 74]

Here, the unit of the zenith angle is radian. Table 3 shows thecoefficients of the third order polynomial.

TABLE 3 variable value a₁ 561.5398 a₂  18.5071 a₃ −30.3594

Due to the experimental measurement error, it is conjectured that thediscrepancy between the approximate projection scheme given by Table 3and the real projection scheme of the lens is larger than 3 pixels.

This algorithm can be verified using a scientific program such as theMatLab. Following is the algorithm for extracting a panoramic imagehaving a horizontal FOV of 180° from the image given in FIG. 29.

% Image processing of a panoramic image. % % *********** Real projectionscheme *************************** coeff = [−30.3594, 18.5071, 561.5398,0.0]; % % *** Read in the graphic image ********** picture =imread(‘image’, ‘jpg’); [Kmax, Lmax, Qmax] = size(picture); CI =double(picture) + 1; % Lo = 1058; % x position of the optical axis inthe raw image Ko = 806; % y position of the optical axis in the rawimage % % Draw an empty canvas Jmax = 1600; % canvas width Imax = 600; %canvas height EI = zeros(Imax, Jmax, 3); % dark canvas % Jo = (1 + Jmax)/ 2; Io = 200; Dpsi = pi; A = (Jmax − 1) / Dpsi; % % Virtual screen forI = 1: Imax for J = 1: Jmax p = J − Jo; q = I − Io; psi = p / A; delta =atan(q / A); phi = atan2(tan(delta), sin(psi)); theta =acos(cos(delta) * cos(psi)); r = polyval(coeff, theta); x = r *cos(phi) + Lo; y = r * sin(phi) + Ko; Km = floor(y); Kp = Km + 1; dK = y− Km; Lm = floor(x); Lp = Lm + 1; dL = x − Lm; if((Km >= 1) & (Kp <=Kmax) & (Lm >= 1) & (Lp <= Lmax)) EI(I, J, :) = (1 − dK) * (1 − dL) *CI(Km, Lm, :) ... + dK*(1 − dL)*CI(Kp, Lm, :) ... + (1 − dK) * dL *CI(Km, Lp, :) ... + dK * dL * CI(Kp, Lp, :); else EI(I, J, :) = zeros(1,1, 3); end end end DI = uint8(EI − 1); imagesc(DI); axis equal;

FIG. 31 is an exemplary panoramic image extracted using this algorithm,where the rectangular processed image plane is 1600 pixels wide in thelateral direction, and 600 pixels high in the longitudinal direction.The coordinate of the origin is (800.5, 200.5), and the horizontal FOVis 180°. Slight errors noticeable in FIG. 31 is mainly due to the errorin aligning the optical axis parallel to the ground plane and the errorin locating the position of the second intersection point.

Second Embodiment

As has been stated previously, a cylindrical projection scheme in thestrict sense of the words is not used widely. Although it provides amathematically most precise panoramic image, the image does not appearnatural to the naked eye when the vertical FOV (i.e., Δδ=δ₂−δ₁) islarge.

The cylindrical projection scheme of the first embodiment can begeneralized as follows. The lateral coordinate x″ on the processed imageplane is proportional to the horizontal incidence angle. Therefore, asin the first embodiment, a relation given in Eq. 75 holds.

$\begin{matrix}{A = {\frac{W}{\Delta \; \psi} = \frac{x^{''}}{\psi}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 75} \right\rbrack\end{matrix}$

On the other hand, the longitudinal coordinate y″ on the processed imageplane is proportional to a monotonic function of the vertical incidenceangle as given in Eq. 76.

y″∝F(δ)  [Math Figure 76]

Here, F(δ) is a continuous and monotonic function of the verticalincidence angle δ. Therefore, a relation given by Eq. 77 correspondingto the Eq. 22 holds as follows.

$\begin{matrix}{A = {\frac{H}{{F\left( \delta_{2} \right)} - {F\left( \delta_{1} \right)}} = \frac{y^{''}}{F(\delta)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 77} \right\rbrack\end{matrix}$

Therefore, the span of the horizontal incidence angle, the span of thevertical incidence angle, and the size of the processed image planesatisfy the following relation.

$\begin{matrix}{\frac{W}{H} = \frac{\Delta \; \psi}{{F\left( \delta_{2} \right)} - {F\left( \delta_{1} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figuure}\mspace{14mu} 78} \right\rbrack\end{matrix}$

Also, the horizontal incidence angle corresponding to the third point onthe processed image plane having a lateral coordinate x″ and alongitudinal coordinate y″ is given by Eq. 79, and the verticalincidence angle is given by Eq. 80.

$\begin{matrix}{\psi = \frac{x^{''}}{A}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 79} \right\rbrack \\{\delta = {F^{- 1}\left( \frac{y^{''}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 80} \right\rbrack\end{matrix}$

Here, F⁻¹ is the inverse function of the function F( ). The saidcylindrical projection scheme in the first embodiment is a case wherethe function F is given by Eq. 81.

F(δ)=tan δ  [Math Figure 81]

On the other hand, if the said general projection scheme is specificallyan equi-rectangular projection scheme, then the said function is givenby Eq. 82.

F(δ)=δ  [Math Figure 82]

Therefore, the ranges of the horizontal and the vertical incidenceangles and the size of the processed image plane satisfy the followingrelation.

$\begin{matrix}{\frac{W}{H} = \frac{\Delta \; W}{\delta_{2} - \delta_{1}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 83} \right\rbrack\end{matrix}$

Also, the vertical incidence angle is given by Eq. 84.

$\begin{matrix}{\delta = \frac{y^{''}}{A}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 84} \right\rbrack\end{matrix}$

On the other hand, if the said general projection scheme is specificallya Mercator projection scheme, then the said function is given by Eq. 85.

$\begin{matrix}{{F(\delta)} = {\ln \left\{ {\tan \left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 85} \right\rbrack\end{matrix}$

Also, the ranges of the horizontal and the vertical incidence angles andthe size of the processed image plane satisfy the following relation.

$\begin{matrix}{\frac{W}{H} = \frac{\Delta \; \psi}{\ln \left\lbrack \frac{\tan \; \left( {\frac{\pi}{4} + \frac{\delta_{2}}{2}} \right)}{\tan \left( {\frac{\pi}{4} + \frac{\delta_{1}}{2}} \right)} \right\rbrack}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 86} \right\rbrack\end{matrix}$

On the other hand, the vertical incidence angle is given by Eq. 87.

$\begin{matrix}{\delta = {{2\; {\tan^{- 1}\left\lbrack {\exp\left( \frac{y^{''}}{A}\; \right)} \right\rbrack}} - \frac{\pi}{2}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 87} \right\rbrack\end{matrix}$

As in the first embodiment of the present invention, considering thefact that the image sensor plane, the uncorrected image plane and theprocessed image plane are all digitized, the image processing methods ofthe first and the second embodiments are comprised of a stage ofacquiring an uncorrected image plane while the optical axis of a cameramounted with a rotationally-symmetric wide-angle lens and the lateralsides of the image sensor plane are aligned parallel to the groundplane, and an image processing stage of extracting a processed imageplane from the uncorrected image plane. Said uncorrected image plane isa two dimensional array having K_(max) rows and L_(max) columns, thepixel coordinate of the optical axis on the said uncorrected image planeis (K_(o), L_(o)), and the real projection scheme of the said lens isgiven as a function given in Eq. 88.

r=r(θ)  [Math Figure 88]

Here, the real projection scheme of the lens refers to the image heightr obtained as a function of the zenith angle θ of the incident ray, andthe magnification ratio g of the said camera is given by Eq. 89, whereinr′ is a pixel distance on the uncorrected image plane corresponding tothe image height r.

$\begin{matrix}{g = \frac{r^{\prime}}{r}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 89} \right\rbrack\end{matrix}$

Said processed image plane is a two dimensional array having I_(max)rows and J_(max) columns, the pixel coordinate of the optical axis onthe processed image plane is (I_(o), J_(o)), and the horizontalincidence angle ψ_(I,J)≡ψ(I, J)=ψ_(J) an incident ray corresponding to apixel having a coordinate (I, J) on the said processed image plane isgiven by Eq. 90 as a sole function of the said pixel coordinate J.

$\begin{matrix}{\psi_{J} = {\frac{\psi_{J\; {ma}\; x} - \psi_{1}}{J_{m\; {ax}} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 90} \right\rbrack\end{matrix}$

Here, ψ¹ is a horizontal incidence angle corresponding to J=1, andψ^(Jmax) is a horizontal incidence angle corresponding to J=J_(max). Onthe other hand, the vertical incidence angle δ_(I,J)∝δ(I, J)=δ_(I) ofthe said incident ray is given by Eq. 91 as a sole function of the saidpixel coordinate I.

$\begin{matrix}{\delta_{I} = {F^{- 1}\left\{ {\frac{\psi_{J\; {ma}\; x} - \psi_{1}}{J_{{ma}\; x} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 91} \right\rbrack\end{matrix}$

Here, F⁻¹ is the inverse function of a continuous and monotonicallyincreasing function F(δ) of the incidence angle δ, and the signal valueof a pixel having a coordinate (I, J) on the said processed image planeis given by the signal value of a virtual pixel having a coordinate(x′_(I,J),y′_(I,J)) on the uncorrected image plane, wherein the saidcoordinate (x′_(I,J),y′_(I,J)) of the virtual pixel is obtained from theseries of equations given in Eqs. 92 through 96.

$\begin{matrix}{\theta_{I,J} = {\cos^{- 1}\left( {\cos \; \delta_{I}\cos \; \psi_{J}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 92} \right\rbrack \\{\varphi_{I,J} = {\tan^{- 1}\left( \frac{\tan \; \delta_{I}}{\sin \; \psi_{J}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 93} \right\rbrack \\{r_{I,J} = {r\left( \theta_{I,J} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 94} \right\rbrack \\{x_{I,J}^{\prime} = {L_{o} + {g\; r_{I,J}\cos \; \varphi_{I,J}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 95} \right\rbrack \\{y_{I,J}^{\prime} = {K_{o} + {g\; r_{I,J}\sin \; \varphi_{I,J}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 96} \right\rbrack\end{matrix}$

If the projection scheme of the panoramic image is a cylindricalprojection scheme, then the said function F is given by Eq. 97, and thesaid vertical incidence angle is given by Eq. 98.

$\begin{matrix}{{F(\delta)} = {\tan \; \delta}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 97} \right\rbrack \\{\delta_{I\;} = {\tan^{- 1}\left\{ {\frac{\psi_{J\; {ma}\; x} - \psi_{1}}{J_{{ma}\; x} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 98} \right\rbrack\end{matrix}$

If the projection scheme of the panoramic image is an equi-rectangularprojection scheme, then the said function F is given by Eq. 99, and thesaid vertical incidence angle is given by Eq. 100.

$\begin{matrix}{{F(\delta)} = \delta} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 99} \right\rbrack \\{\delta_{l} = {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 100} \right\rbrack\end{matrix}$

If the projection scheme of the panoramic image is a Mercator projectionscheme, then the said function F is given by Eq. 101, and the saidvertical incidence angle is given by Eq. 102.

$\begin{matrix}{{F(\delta)} = {\ln \left\{ {\tan \left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 101} \right\rbrack \\{\delta_{l} = {{2{\tan^{- 1}\left\lbrack {\exp \left\{ {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}} \right\rbrack}} - \frac{\pi}{2}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 102} \right\rbrack\end{matrix}$

The procedure for finding the location of the image point correspondingto given horizontal and vertical incidence angles and interpolating theimage is identical to the procedure given in Eqs. 63 through 71.

FIG. 32 is a panoramic image following an equi-rectangular projectionscheme extracted from the fisheye image given in FIG. 26, and FIG. 33 isa panoramic image following a Mercator projection scheme. Especially inthe panoramic image in FIG. 32 with an equi-rectangular projectionscheme, the horizontal and the vertical FOVs are both 180°.

Third Embodiment

Field of view of typical wide-angle lenses used in security andsurveillance area is 90° at the maximum, and a security lens having thisamount of FOV generally exhibits a considerable degree of distortion. Alens having a FOV larger than this is not widely used because such alens exhibits excessive distortion and causes psychological discomfort.

FIG. 34 is a conceptual drawing of an imaging system(3400) formonitoring the entire interior space using a typical wide-angle camerahaving a FOV near 90°. Usually, security cameras must be installed at ahigh place that is out of reach from the people. Therefore, in order tomonitor the entire interior place while kept out of reach from thepeople, the camera(3410) is installed at a corner where the ceiling andtwo walls meet, and camera is facing down toward the interior. Theoptical axis(3401) of the camera is inclined both against a horizontalline and a vertical line, and the camera captures the objects(3467) at aslanted angle. When such a camera with a FOV near 90° is used, theregion(3463) captured by the camera can contain the entire interiorspace, but there is a large difference in distances between the closerand the further sides, and visual information for the further side canbe insufficient.

Usually, lighting(3471) is installed at an interior ceiling. Sincetypical wide-angle camera is installed facing down toward the interiorat an angle, the said lighting is out of the field of view(3463) of thecamera, and the intense rays(3473) from the lighting are not captured bythe camera.

FIG. 35 is a diagram schematically illustrating the desirable size andthe location of real image(3533) on an image sensor plane(3513) in thethird embodiment of the present invention. The fisheye lens suitable forthe present embodiment is a fisheye lens with a FOV greater than 180°and following an equidistance projection scheme. Furthermore, theintersection point O between the optical axis of the fisheye lens andthe image sensor plane is located higher than the very center of theimage sensor plane. If the maximum FOV of this fisheye lens is given as2θ₂, then the image height of an incident ray at the image sensor planehaving the maximum zenith angle is given as r₂=r(θ₂). Here, thedesirable image height is given by Eq. 73. Desirably, the imagecicle(3533) contacts the left edge(3513L), the right edge(3513R) and thebottom edge(3513B) of the image sensor plane(3513). In thisconfiguration, the imaging system uses the most out of the pixels on theimage sensor plane and monitors the entire interior space whileinstalled near the ceiling of the interior space.

FIG. 36 is a conceptual drawing of a desirable installation state of animaging system(3600) using an ultra wide-angle lens with a FOV largerthan those of typical wide-angle lenses for the purpose of monitoringthe entire interior space. If an ultra wide-angle lens with a FOV muchlarger than those of typical wide-angle lenses is used, then the cameraneeds not be installed at a slanted angle as in FIG. 34 in order tomonitor the entire interior space.

In order to acquire a natural-looking panoramic image, the opticalaxis(3601) of the image acquisition means(3610) must be parallel to theground plane. Also, the security camera must be installed at a highplace that is out of reach from the people. Therefore, as schematicallyillustrated in FIG. 36, the camera must be installed at a high placebelow the ceiling with the optical axis aligned parallel to the groundplane. However, if the camera is installed in this way, then typicallythe half of the screen will be occupied by the ceiling, and the areabelow the ceiling that has to be monitored occupies only a portion ofthe screen. Therefore, through an arrangement of the image sensor planeshown in FIG. 35, the vertical FOV needs to be made asymmetrical.Furthermore, since the camera is installed horizontally, thelighting(3671) can be contained within the field of view(3663) of thecamera. Since the image cannot be not satisfactory when such intenserays are captured by the camera, a cover(3675) blocking the region abovethe horizon as illustrated in FIG. 36 can be useful.

Under this physical environment, a panoramic image with a horizontal FOVaround 180° can be obtained such as the ones shown in the first and thesecond embodiments of the present invention, while the vertical FOV ismade asymmetrical. Also, a cylindrical projection scheme is desirablewhen the vertical FOV is near the standard FOV around 60°, and aMercator projection scheme is desirable when the FOV is larger. Also, inan application example where any dead zone in the security monitoringeither in the ceiling or the floor cannot be tolerated, then anequi-rectangular projection scheme can be used. In this case, thehorizontal FOV as well as the vertical FOV will be both 180°.

Fourth Embodiment

FIG. 37 is a schematic diagram of an ordinary car rear viewcamera(3710). For a car rear view camera, it is rather common that awide-angle lens with more than 150° FOV is used, and the optical axis ofthe lens is typically inclined toward the ground plane(3717) asillustrated in FIG. 37. By installing the camera in this way, parkinglane can be easily recognized when backing up the car. Furthermore,since the lens surface is oriented downward toward the ground,precipitation of dust is prevented, and partial protection is providedfrom rain and snow.

It is desirable to install the image acquisition means(3710) of thefourth embodiment of the present invention on top of the trunk of apassenger car, and to align the optical axis at a certain angle with theground plane. Furthermore, a fisheye lens with a FOV larger than 180°and following an equidistance projection scheme is most preferable, andthe image display means is desirably installed next to the driver seat.

Using a wide-angle camera with its optical axis inclined toward theground plane, it is possible to obtain a panoramic image such as thoseshown in the first and the second embodiments of the present invention.The world coordinate system of this embodiment takes the nodal point Nof the imaging system(3710) as the origin, and takes a vertical linethat is perpendicular to the ground plane(3717) as the Y-axis, and theZ-axis is set parallel to the car(3751) axle. According to theconvention of right handed coordinate system, the positive direction ofthe X-axis is the direction directly plunging into the paper in FIG. 37.Therefore, if the lens optical axis is inclined below the horizon withan angle α, then a coordinate system fixed to the camera has beenrotated around the X-axis of the world coordinate system by angle α.This coordinate system is referred to as the first world coordinatesystem, and the three axes of this first world coordinate system arenamed as X′, Y′ and Z′-axis, respectively. In FIG. 37, it appears thatthe first world coordinate system has been rotated around the X-axisclockwise by angle α relative to the world coordinate system. However,considering the direction of the positive X-axis, it has been in factrotated counterclockwise by angle α. Since direction of rotationconsiders counterclockwise rotation as the positive direction, the firstworld coordinate system in FIG. 37 has been rotated by +α around theX-axis of the world coordinate system.

Regarding the rotation of coordinate system, it is convenient to use theEuler matrices. For this, the coordinate of an object point Q in athree-dimensional space is designated as a three-dimensional vector asgiven below.

$\begin{matrix}{\overset{\_}{Q} = \begin{pmatrix}X \\Y \\Z\end{pmatrix}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 103} \right\rbrack\end{matrix}$

Here, Q represents a vector in the world coordinate system starting atthe origin and ending at the point Q in the three-dimensional space.Then, the coordinate of a new point obtainable by rotating the point Qin the space by an angle −α around the X-axis is given by multiplyingthe matrix given in Eq. 104 on the above vector.

$\begin{matrix}{{M_{X}(\alpha)} = \begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \alpha} & {\sin \; \alpha} \\0 & {{- \sin}\; \alpha} & {\cos \; \alpha}\end{pmatrix}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 104} \right\rbrack\end{matrix}$

Likewise, the matrix given in Eq. 105 can be used to find the coordinateof a new point which is obtainable by rotating the point Q by angle −βaround the Y-axis, and the matrix given in Eq. 106 can be used to findthe coordinate of a new point which is obtainable by rotating the pointQ by angle −γ around the Z-axis.

$\begin{matrix}{{M_{Y}(\beta)} = \begin{pmatrix}{\cos \; \beta} & 0 & {{- \sin}\; \beta} \\0 & 1 & 0 \\{\sin \; \beta} & 0 & {\cos \; \beta}\end{pmatrix}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 105} \right\rbrack \\{{M_{Z}(\gamma)} = \begin{pmatrix}{\cos \; \gamma} & {\sin \; \gamma} & 0 \\{{- \sin}\; \gamma} & {\cos \; \gamma} & 0 \\{0~} & 0 & 1\end{pmatrix}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 106} \right\rbrack\end{matrix}$

Matrices in Eqs. 104 through 106 can describe the case where thecoordinate system is fixed and the point in space has been rotated, butalso the same matrices can describe the case where the point in space isfixed and the coordinate system has been rotated in the reversedirection. These two cases are mathematically equivalent. Therefore, thecoordinate of a point Q in the first world coordinate system that isobtained by rotating the world coordinate system by angle α around theX-axis as indicated in FIG. 37 is given by Eq. 107.

$\begin{matrix}{{\overset{\_}{Q}}^{\prime} = {\begin{pmatrix}X^{\prime} \\Y^{\prime} \\Z^{\prime}\end{pmatrix} = {{M_{X}(\alpha)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 107} \right\rbrack\end{matrix}$

Using the matrix given in Eq. 104, the coordinate in the first worldcoordinate system can be given as follows in terms of the coordinate inthe world coordinate system.

X′=X  [Math Figure 108]

Y′=Y cos α+Z sin α  [Math Figure 109]

Z′=−Y sin α+Z cos α  [Math Figure 110]

Referring to FIG. 37, let's assume that an imaging system has beeninstalled with its optical axis inclined toward the ground plane, andnevertheless, it is desired to obtain a panoramic image that is parallelto the ground plane. FIG. 38 is a wide-angle image with α=30°. In thiscase, the following algorithm can be used to obtain a panoramic imagethat is parallel to the ground plane. First, under the assumption thatthe said imaging system is parallel to the ground plane, the size of theprocessed image plane, the location of the intersection point, and thefield of view are determined by the same method as in the examples ofthe previous embodiments. Then, the horizontal incidence anglecorresponding to the lateral coordinate x″ on the processed image planeand the vertical incidence angle corresponding to the longitudinalcoordinate y″ are given by Eqs. 111 through 113.

$\begin{matrix}{\psi = \frac{x^{''}}{A}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 111} \right\rbrack \\{A = \frac{W}{\Delta \; \psi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 112} \right\rbrack \\{\delta = {F^{- 1}\left( \frac{y^{''}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 113} \right\rbrack\end{matrix}$

Next, it is assumed that an incident ray having these horizontal andvertical incidence angles has been originated from an object point on ahemisphere with a radius 1 and having its center at the nodal point ofthe lens. Then, the coordinate of the said object point in the worldcoordinate system is given by Eqs. 114 through 116.

X=cos δ sin ψ  [Math Figure 114]

Y=sin δ  [Math Figure 115]

Z=cos δ cos ψ  [Math Figure 116]

The coordinate of this object point in the first world coordinate systemis given by Eqs. 108 through 110. The X′, Y′ and Z′-axes of this firstworld coordinate system are parallel to the x, y, and z-axes of thefirst rectangular coordinate system, respectively. Therefore, the zenithand the azimuth angles of the incident ray are given by Eqs. 117 and 118by the same method in the first embodiment.

$\begin{matrix}{\theta = {\cos^{- 1}\left( Z^{\prime} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 117} \right\rbrack \\{\varphi = {\tan^{- 1}\left( \frac{Y^{\prime}}{X^{\prime}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 118} \right\rbrack\end{matrix}$

Finally, the location of the first point on the image sensor planehaving these zenith and azimuth angles can be obtained by the samemethods as in the first and the second embodiments.

Identical to the previous examples, considering the fact that all theimage sensors and display devices are digital devices, image processingprocedure must use the following set of equations. After a series ofpreparatory stages have been taken as in the first and the secondembodiments, the desirable size of the processed image plane and thelocation (I_(o), J_(o)) of the third intersection point are determined,and then the horizontal incidence angle ψ_(J) and the vertical incidenceangle δ_(I) given by Eqs. 119 and 120 are computed for all the pixels(I, J) on the said processed image plane.

$\begin{matrix}{\psi_{J} = {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 119} \right\rbrack \\{\delta_{I} = {F^{- 1}\left\{ {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 120} \right\rbrack\end{matrix}$

From these horizontal and vertical incidence angles, the coordinate ofan imaginary object point in the world coordinate system is calculatedusing Eqs. 121 through 123.

X_(I,J)=cos δ_(I) sin ψ_(J)  [Math Figure 121]

Y_(I,J)=sin δ_(I)  [Math Figure 122]

Z_(I,J)=cos δ_(I) cos ψ_(J)  [Math Figure 123]

From this coordinate of the object point in the world coordinate system,the coordinate of the object point in the first world coordinate systemis obtained using Eqs. 124 through 126.

X′_(I,J)=X_(I,J)  [Math Figure 124]

Y′ _(I,J) =Y _(I,J) cos α+Z _(I,J) sin α  [Math Figure 125]

Z′ _(I,J) =−Y _(I,J) sin α+Z _(I,J) cos α  [Math Figure 126]

From this coordinate, the zenith angle θ_(I,J) and the azimuth angleΦ_(I,J) of the incident ray are computed using Eqs. 127 and 128.

$\begin{matrix}{\theta_{I,J} = {\cos^{- 1}\left( Z_{I,J}^{\prime} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 127} \right\rbrack \\{\varphi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}^{\prime}}{X_{I,J}^{\prime}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 128} \right\rbrack\end{matrix}$

Next, the image height r_(u) on the image sensor plane is calculatedusing Eq. 129.

r _(I,J) =r(θ_(I,J))  [Math Figure 129]

Then, the position (K_(o), L_(o)) of the second intersection point onthe uncorrected image plane and the magnification ratio g are used tofind the position of the second point on the uncorrected image plane.

x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Math Figure 130]

y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Math Figure 131]

Once the position of the corresponding second point has been found, theninterpolation methods such as described in the first and the secondembodiments can be used to obtain a panoramic image.

FIG. 39 is a panoramic image obtained using this method where acylindrical projection scheme has been employed. As can be seen fromFIG. 39, a panoramic image identical to that in the first embodiment hasbeen obtained despite the fact that the optical axis is not parallel tothe ground plane. Using such a panoramic imaging system as a car rearview camera, the backside of a vehicle can be entirely monitored withoutany dead spot.

One point which needs special attention when using such an imagingsystem as a car rear view camera is the fact that for a device (i.e., acar) of which the moving direction is the exact opposite of the opticalaxis direction of the image acquisition means, it can cause a greatconfusion to the driver if a panoramic image obtained by the methods inthe first through the fourth embodiments is displayed without anyfurther processing. Since a car rear view camera is heading toward thebackside of the car, the right end of the car appears as the left end ona monitor showing the images captured by the rear view camera. However,the driver can fool himself by thinking that the image is showing theleft end of the car from his own viewpoint of looking at the front endof the car, and thus, there is a great danger of possible accidents. Toprevent such a perilous confusion, it is important to switch the leftand the right sides of the image obtained using a car rear view camerabefore displaying it on the monitor. The video signal S′(I, J) for thepixel in the mirrored (i.e., the left and the right sides are exchanged)processed image plane with a coordinate (I, J) is given by the videosignal S(I, J_(max)−J+1) from the pixel in the processed image planewith a coordinate (I, J_(max)−J+1).

S′(I,J)=S(I,J _(max) −J+1)  [Math Figure 132]

On the other hand, an identical system can be installed near the roommirror, frontal bumper, or the radiator grill in order to be used as arecording camera connected to a car black box for the purpose ofrecording vehicle's driving history.

Above embodiment has been described in relation to a car rear viewcamera, but it must be obvious that the usefulness of the inventiondescribed in this embodiment is not limited to a car rear view camera.

Fifth Embodiment

For large buses and trucks, it is necessary to monitor the lateral sidesof the vehicle as well as the rear side of the vehicle. Such a sidemonitoring imaging system will be especially useful when making a turnin a narrow alley, or when changing lanes in a highway. Also, it will beuseful as a security-monitoring camera for the purpose of preventingaccidents when passengers are boarding on the bus or getting off fromthe bus. FIG. 40 is a conceptual drawing of such a device, and it showsan aerial view of the device seen from above. When image acquisitionmeans of the present invention, in other words, video cameras equippedwith an equidistance projection fisheye lens with 180° FOV, areinstalled near the top of the front, the rear, and the two side walls ofthe said vehicle(4051), then each camera monitors 180° on theirrespective walls without any dead spot, and as a whole, the entiresurroundings of the vehicle can be monitored without any dead zone.Also, by installing cameras in the same way on the outer walls of abuilding, all the directions can be monitored without any dead zone. Theoptical axes of the cameras can be parallel to the ground plane as inthe first through the third embodiments of the present invention, or itcan be slanted at an angle as in the fourth embodiment.

Sixth Embodiment

FIG. 41 illustrates an application example where the longitudinal sideof the image sensor plane may not be perpendicular to the ground plane.If a wide-angle camera(4110) is installed on a motorcycle(4151) shown inFIG. 41 for the purpose of recording the driving history, thelongitudinal side of the image sensor plane can happen to be notperpendicular to the ground plane(4117). Illustrated in FIG. 41 is acase where the X-axis of the world coordinate system that is fixed tothe ground plane and the x-axis of the first rectangular coordinatesystem that is fixed to the motorcycle differ by an angle γ. Especially,if the motorcycle has to change its moving direction, then themotorcycle has to be inclined toward that direction, and therefore, sucha situation inevitably occurs. Similar situations occur for a shipfloating in a sea with high waves, or for an airplane or an UAV(unmanned aerial vehicle) performing acrobatic flights. Also, identicalsituation can occur for a vehicle on an excessively slanted road.

FIG. 42 is an imaginary fisheye image acquired using an imaging system,whereof the longitudinal sides of the image sensor plane are slanted by20° with respect to a vertical line. On the other hand, FIG. 43 is apanoramic image following a cylindrical projection scheme described inthe first embodiment extracted from the image given in FIG. 42. As canbe seen from FIG. 43, the image appears quite unnatural. This is becausethe imaging systems in the first and the second embodiments are imagingsystems that capture straight lines parallel to the longitudinal sidesof the image sensor plane as straight lines, and therefore, desirablepanoramic image cannot be obtained if a vertical line and thelongitudinal sides of the image sensor plane are not parallel to eachother.

FIG. 44 is a diagram schematically illustrating the desirable size andthe location of real image(4433) on the image sensor plane(4413)according to the sixth embodiment of the present invention. The fisheyelens suitable for the present embodiment is a fisheye lens with a FOVgreater than 180° and following an equidistance projection scheme.Furthermore, the intersection point O between the optical axis of thefisheye lens and the image sensor plane is located at the center of theimage sensor plane. Therefore, the range of the lateral coordinate is(−B/2≦x≦B/2), and the range of the longitudinal coordinate is(−V/2≦y≦V/2).

If the maximum FOV of this fisheye lens is given as 2θ₂, then the imageheight of an incident ray on the image sensor plane having the maximumzenith angle is given as r₂≡r(θ₂). Here, the desirable image height isgiven by Eq. 133.

$\begin{matrix}{{r\left( \theta_{2} \right)} = \frac{V}{2}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 133} \right\rbrack\end{matrix}$

Therefore, the image circle(4433) contacts the top edge(4413T) and thebottom edge(4413B) of the image sensor plane(4413). In thisconfiguration, always the same horizontal FOV can be obtained even ifthe imaging system is slanted at an arbitrary angle with respect to theground plane.

FIG. 45 is a schematic diagram of the sixth embodiment of the presentinvention, which is a device that mainly includes an image acquisitionmeans(4510), an image processing means(4516), image display means(4517)as well as a direction sensing means(4561). Said direction sensingmeans(4561) indicates the slanting angle γ between the first rectangularcoordinate system describing the said image acquisition means and theworld coordinate system describing the objects around the imagingsystem. Here, the angle γ is the slanting angle between the X-axis ofthe world coordinate system and the x-axis of the first rectangularcoordinate system, or the slanting angle between the Y-axis of the worldcoordinate system and the minus(−) y-axis of the first rectangularcoordinate system. Since said direction sensing means are widespread asto be embedded in most of the camera phones and digital cameras, adetailed description will be omitted. The device of the sixth embodimentof the present invention is characterized in that it provides apanoramic image referencing on the Y-axis of the world coordinate systemusing the angle γ obtained from the said direction sensing means.

Referring to FIG. 46, the third point P″ on the processed imageplane(4635) has a rectangular coordinate (x″, y″). Incidentally, if thelongitudinal sides of the processed image plane is parallel to theY-axis of the world coordinate system, then the coordinate of this thirdpoint will be comprised of a lateral coordinate x′″ given by Eq. 134 anda longitudinal coordinate y′″ given by Eq. 135.

x′″=x″ cos γ+y″ sin γ  [Math Figure 134]

y′″=−x″ sin γ+y″ cos γ  [Math Figure 135]

Therefore, the horizontal and the vertical incidence angles of anincident ray corresponding to this third point must be given by Eq. 136and Eq. 137, respectively.

$\begin{matrix}{\psi = {{\frac{\Delta\psi}{W}x^{\prime\prime\prime}} = \frac{x^{\prime\prime\prime}}{A}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 136} \right\rbrack \\{\delta = {F^{- 1}\left( \frac{y^{\prime\prime\prime}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 137} \right\rbrack\end{matrix}$

Therefore, the signal value of the third point on the processed imageplane having an ideal projection scheme must be the signal value of animage point on the image sensor plane formed by an incident rayoriginated from an object point on the object plane having a horizontalincidence angle (i.e., the longitude) given by Eq. 136 and a verticalincidence angle (i.e., the latitude) given by Eq. 137. The zenith angleof this incident ray is given by Eq. 138, the azimuth angle is given byEq. 139, and the image height is given by Eq. 140.

$\begin{matrix}{\theta = {\cos^{- 1}\left( {\cos \; \delta \; \cos \; \psi} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 138} \right\rbrack \\{\varphi = {\tan^{- 1}\left( \frac{\tan \; \delta}{\sin \; \psi} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 139} \right\rbrack \\{r = {r(\theta)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 140} \right\rbrack\end{matrix}$

The image point corresponding to an object point having these zenith andazimuth angles has a two-dimensional rectangular coordinate given byEqs. 141 and 142 in the second rectangular coordinate system, which is acoordinate system with the axes rotated by angle γ with respect to theY-axis.

x′=gr(θ)cos(γ+φ)  [Math Figure 141]

y′=gr(θ)sin(γ+φ)  [Math Figure 142]

Therefore, it suffice to assign the signal value of an image point onthe uncorrected image plane having this rectangular coordinate as thesignal value of the third point on the processed image plane.

Identical to the previous examples, considering the fact that all theimage sensors and display devices are digital devices, image processingprocedure must use the following set of equations. After a series ofpreparatory stages have been taken as in the first through the fifthembodiments, the desirable size of the processed image plane and thelocation (I_(o), J_(o)) of the third intersection point are determined,and then the horizontal incidence angle ψ_(I,J) and the verticalincidence angle δ_(I,J) given by Eqs. 143 and 144 are computed for allthe pixels (I, J) on the said processed image plane.

$\begin{matrix}{\mspace{79mu} {\psi_{I,J} = {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left\{ {{\left( {J - J_{o}} \right)\cos \; \gamma} + {\left( {I - I_{o}} \right)\sin \; \gamma}} \right\}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 143} \right\rbrack \\{\delta_{I,J} = {F^{- 1}\left\lbrack {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left\{ {{{- \left( {J - J_{o}} \right)}\sin \; \gamma} + {\left( {I - I_{o}} \right)\cos \; \gamma}} \right\}} \right\rbrack}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 144} \right\rbrack\end{matrix}$

From these horizontal and vertical incidence angles, the zenith angleθ_(I,J) and the azimuth angle Φ_(I,J) of an incident ray in the worldcoordinate system are calculated using Eqs. 145 and 146.

$\begin{matrix}{\theta_{I,J} = {\cos^{- 1}\left( {\cos \; \delta_{I,J}\cos \; \psi_{I,J}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 145} \right\rbrack \\{\varphi_{I,J} = {\tan^{- 1}\left( \frac{\tan \; \delta_{I,J}}{\sin \; \psi_{I,J}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 146} \right\rbrack\end{matrix}$

Next, the image height r_(I,J) on the image sensor plane is calculatedusing Eq. 147.

r _(I,J) =r(θ_(I,J))  [Math Figure 147]

Next, the position (K_(o), L_(o)) of the second intersection point onthe uncorrected image plane and the magnification ratio g are used tofind the position of the second point on the uncorrected image plane.

x′ _(I,J) =L _(o) +gr _(I,J) cos(γ+φ_(I,J))  [Math Figure 148]

y′ _(I,J) =K _(o) +gr _(I,J) sin(γ+φ_(I,J))  [Math Figure 149]

Once the position of the corresponding second point has been found, theninterpolation methods such as described in the first through the thirdembodiments can be used to obtain a panoramic image.

The desirable examples of the monotonically increasing function F(δ) ofthe incidence angle in this embodiment can be given by Eqs. 97, 99 and101. FIG. 47 is a panoramic image acquired using this method, and acylindrical projection scheme given in Eq. 97 has been used. As can beseen from FIG. 47, vertical lines appear as straight lines slanted at anangle γ. Such an imaging system provides a satisfactory panoramic imageas well as an accurate feedback about the vertical direction.

Seventh Embodiment

When an airplane takes off from the ground, or when it is landing down,or when it is making a turn, the airplane body is leaning sideways aswell as toward the moving direction. FIGS. 48 and 49 illustrate such anapplication example. The imaging system is assumed as has been installedparallel to the airplane body. The world coordinate system of thepresent embodiment takes the nodal point of the imaging system as theorigin, and takes a vertical line that is perpendicular to the groundplane as the Y-axis, and the direction the airplane is heading while theairplane is maintaining its body in a horizontal posture is thedirection of the Z-axis. Incidentally, if the airplane is leaningforward by an angle α, then a coordinate system fixed to the airplanehas been rotated by angle α around the X-axis of the world coordinatesystem. This coordinate system is the first world coordinate system, andthe three axes of this first world coordinate system are referred to asX′, Y′, and Z′-axis, respectively. On the other hand, if the airplanehas been also inclined laterally by angle γ as has been illustrated inFIG. 49, then a coordinate system fixed to the airplane has been rotatedby angle γ around the Z′-axis of the first world coordinate system. Thiscoordinate system is called as the second world coordinate system, andthe three axes of this second world coordinate system are referred to asX″, Y″, and Z″-axis, respectively.

Regarding the rotation of coordinate system, it is convenient to use theEuler matrices as in the fourth embodiment. The coordinate of the saidone point in the first world coordinate system, which is a coordinatesystem that has been rotated by angle α around the X-axis as shown inFIG. 48, is given by Eq. 150.

$\begin{matrix}{{\overset{\rightarrow}{Q}}^{\prime} = {\begin{pmatrix}X^{\prime} \\Y^{\prime} \\Z^{\prime}\end{pmatrix} = {{M_{X}(\alpha)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 150} \right\rbrack\end{matrix}$

On the other hand, referring to FIG. 49, the second world coordinatesystem is the first world coordinate system that has been rotated byangle γ around the Z′-axis of the first world coordinate system.Therefore, the coordinate of the said one point in the second worldcoordinate system is given by Eq. 151.

$\begin{matrix}{{\overset{\rightarrow}{Q}}^{''} = {\begin{pmatrix}X^{''} \\Y^{''} \\Z^{''}\end{pmatrix} = {{{M_{Z^{\prime}}(\gamma)}{\overset{\rightarrow}{Q}}^{\prime}} = {{M_{Z^{\prime}}(\gamma)}{M_{X}(\alpha)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 151} \right\rbrack\end{matrix}$

Here, a rotational operation by angle γ around the Z′-axis is acompletely different operation from a rotational operation by angle γaround the Z-axis. However, using the Euler matrices, a rotationalmatrix referring to the axes in the first world coordinate system can bewritten in terms of rotational matrices in the world coordinate system.

M _(Z′)(γ)=M _(X)(α)M _(Z)(γ)M _(X)(−α)  [Math Figure 152]

Therefore, Eq. 151 can be simplified as follows.

$\begin{matrix}{\begin{pmatrix}X^{''} \\Y^{''} \\Z^{''}\end{pmatrix} = {{M_{X}(\alpha)}{M_{Z}(\gamma)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 153} \right\rbrack\end{matrix}$

Using the rotational matrices given in Eqs. 104 and 106, the coordinatein the second world coordinate system can be written in terms of thecoordinate in the world coordinate system as follows.

X″=X cos γ+Y sin γ  [Math Figure 154]

Y″=−X cos α sin γ+Y cos α cos γ+Z sin α  [Math Figure 155]

Z″=X sin α sin γ−Y sin α cos γ+Z cos α  [Math Figure 156]

Referring to FIGS. 48 and 49, let's assume that an imaging system hasbeen installed parallel to the airplane body, and let's further assumethat it is desired to obtain a panoramic image that is parallel to theground plane, irrespective of the inclination of the airplane to theground plane. Such a configuration can be useful when it is required tomonitor the area around the horizon at all times, including the timewhen the airplane is taking off from the ground, landing down, or makinga turn. Also, the same demands may exist for motorcycles or ships, andit may be necessary to monitor the area around the horizon at all timesby military purposes, such as by interceptor missiles.

If a wide-angle imaging system is installed parallel to the body of adevice such as an airplane and a ship as illustrated in FIG. 48, then apanoramic image can not be obtained from an image acquired using such afisheye lens by the methods described in the first through the sixthembodiments of the present invention. FIG. 50 shows an exemplarywide-angle image when the leaning angles are given as α=30° and γ=40°.

In this case, the following algorithm can be used to obtain a panoramicimage that is parallel to the ground plane irrespective of theinclination angles of the device. First, such a system needs a directionsensing means as in the sixth embodiment, and this direction sensingmeans must provide two angular values α and γ to the image processingmeans.

Then, under the assumption that the said imaging system is parallel tothe ground plane, the size of the processed image plane, the location ofthe intersection point, and the field of view are determined by the samemethod as in the examples of the previous embodiments. Therefore, thehorizontal incidence angle corresponding to the lateral coordinate x″ onthe processed image plane and the vertical incidence angle correspondingto the longitudinal coordinate y″ are given by Eqs. 157 through 159.

$\begin{matrix}{\psi = \frac{x^{''}}{A}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 157} \right\rbrack \\{A = \frac{W}{\Delta \; \psi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 158} \right\rbrack \\{\delta = {F^{- 1}\left( \frac{y^{''}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 159} \right\rbrack\end{matrix}$

Next, it is assumed that an incident ray having these horizontal andvertical incidence angles has been originated from an object point on ahemisphere with a radius 1 and having its center at the nodal point ofthe said lens. Then, the coordinate of the said object point in theworld coordinate system is given by Eqs. 160 through 162.

X=cos δ sin ψ  [Math Figure 160]

Y=sin δ  [Math Figure 161]

Z=cos δ cos ψ  [Math Figure 162]

The coordinate of this object point in the second world coordinatesystem is given by Eqs. 154 through 156. The X″, Y″ and Z″-axes of thissecond world coordinate system are parallel to the x, y, and z-axes ofthe first rectangular coordinate system, respectively. Therefore, thezenith and the azimuth angles of the incident ray are given by Eqs. 163and 164 by the same method of the first embodiment.

θ=cos⁻¹(Z″)  [Math Figure 163]

$\begin{matrix}{\varphi = {\tan^{- 1}\left( \frac{Y^{''}}{X^{''}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 164} \right\rbrack\end{matrix}$

Finally, the location of the first point on the image sensor planehaving these zenith and azimuth angles can be obtained by the samemethods as in the first through the fourth embodiments.

Identical to the previous examples, considering the fact that all theimage sensors and display devices are digital devices, image processingprocedure must use the following set of equations. After a series ofpreparatory stages have been taken as in the first through the fourthembodiments, the desirable size of the processed image plane and thelocation (I_(o), J_(o)) of the third intersection point are determined,and then the horizontal incidence angle ψ_(J) and the vertical incidenceangle δ_(I) given by Eqs. 165 and 166 are computed for all the pixels(I, J) on the said processed image plane.

$\begin{matrix}{\psi_{J} = {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 165} \right\rbrack \\{\delta_{I} = {F^{- 1}\left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 166} \right\rbrack\end{matrix}$

From these horizontal and vertical incidence angles, the coordinate ofthe imaginary object point in the world coordinate system is calculatedusing Eqs. 167 through 169.

X_(I,J)=cos δ_(I) sin ψ_(J)  [Math Figure 167]

Y_(I,J)=sin δ_(I)  [Math Figure 168]

Z_(I,J)=cos δ_(I) cos ψ_(J)  [Math Figure 169]

From this coordinate of the object point in the world coordinate system,the coordinate of the object point in the second world coordinate systemis obtained using Eqs. 170 through 172.

X″ _(I,J) =X _(I,J) cos γ+Y _(I,J) sin γ  [Math Figure 170]

Y″ _(I,J) =−X _(I,J) cos α sin γ+Y _(I,J) cos α cos γ+Z _(I,J) sinα  [Math Figure 171]

Z″ _(I,J) =X _(I,J) sin α sin γ−Y _(I,J) sin α cos γ+Z _(I,J) cosα  [Math Figure 172]

From this coordinate, the zenith angle θ_(I,J) and the azimuth angleΦ_(I,J) of the incident ray, are computed using Eqs. 173 and 174.

θ_(I,J)=cos⁻¹(Z″ _(I,J))  [Math Figure 173]

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}^{''}}{X_{I,J}^{''}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 174} \right\rbrack\end{matrix}$

Next, the image height r_(I,J) on the image sensor plane is calculatedusing Eq. 175.

r _(I,J) =r(θ_(I,J))  [Math Figure 175]

Then, the position (K_(o), L_(o)) of the second intersection point onthe uncorrected image plane and the magnification ratio g are used tofind the position of the second point on the uncorrected image plane.

x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Math Figure 176]

y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Math Figure 177]

Once the position of the corresponding first point has been found, theninterpolation methods such as described in the first and the secondembodiments can be used to obtain a panoramic image.

The desirable examples of the monotonically increasing function F(δ) ofthe incidence angle in this embodiment can be given by Eqs. 97, 99 and101. FIG. 51 is a panoramic image acquired using this method, and aMercator projection scheme given in Eq. 101 has been used. As can beseen from FIG. 51, vertical lines all appear as vertical lines.

It can be seen that the fourth embodiment is a special case of theseventh embodiment. In other words, the fourth embodiment is the seventhembodiment with a constraint of γ=0°.

Eighth Embodiment

FIG. 52 is a conceptual drawing of an object plane according to theeighth embodiment of the present invention. In this embodiment, awide-angle lens is used to obtain the views of every 360° direction froman observer, and it is preferable that the optical axis(5201) of thewide-angle lens is perpendicular to the ground plane. Furthermore, allkind of wide-angle lenses that are rotationally symmetric about opticalaxes can be used, which include a refractive fisheye lens with astereographic projection scheme shown in FIG. 53, a catadioptric fisheyelens with a stereographic projection scheme shown in FIG. 54, and acatadioptric panoramic lens schematically illustrated in FIG. 1, as wellas a fisheye lens with an equidistance projection scheme.

In this embodiment, a celestial sphere(5230) with a radius S is assumedthat takes the nodal point N of the lens as the center of the sphere.When all the points(5209) having a latitude angle x from the groundplane(5217), i.e., the X-Y plane, are marked, then the collection ofthese points form a small circle(5239) on the celestial sphere. A coneis further assumed, whereof the tangential points with this celestialsphere comprise the said small circle. Then, the vertex half-angle ofthis cone is also x, and the rotational symmetry axis of the conecoincides with the Z-axis. Hereinafter, this vertex half-angle isreferred to as the reference angle.

FIG. 55 shows a cross-section of the said cone in an incidence planecontaining the Z-axis. An object point(5504) on the said cone has anelevation angle μ with respect to a line segment connecting the nodalpoint N of the lens and the said tangential point(5509). The panoramicimaging system of the present embodiment is an imaging system taking apart of the said cone having an elevation angle between μ₁ and μ₂ withrespect to the said tangential point as an object plane(5531).

The elevation angle and the azimuth angle of an incident ray originatingfrom the said object point(5504) on the object plane can be obtained bythe following method. Identical to the first embodiment, a vector fromthe origin N in the world coordinate system to the said objectpoint(5504) on the object plane can be written a R. The direction ofthis vector is the exact opposite of the propagation direction of theincident ray, and this vector can be written in the world coordinatesystem as in Eq. 178.

R=R{circumflex over (R)}(θ,φ)=X{circumflex over (X)}+YŶ+Z{circumflexover (Z)}  [Math Figure 178]

In Eq. 178, {circumflex over (X)}=(1,0,0) is the unit vector along theX-axis direction, and likewise, Ŷ=(0,1,0) and {circumflex over(Z)}=(0,0,1) are the unit vectors along the Y-axis and the Z-axisdirections, respectively, and {circumflex over (R)} is the directionvector of the said vector, and R is the size of the said vector. Then,the following relations hold between the rectangular coordinate and thepolar coordinate.

X={circumflex over (X)}· R=R sin θ cos φ  [Math Figure 179]

Y=Ŷ· R=R sin θ sin φ  [Math Figure 180]

Z={circumflex over (Z)}· R=R cos θ  [Math Figure 181]

Therefore, using Eqs. 179 through 181, the zenith angle θ and theazimuth angle φ of the incident ray can be obtained from the rectangularcoordinate (X, Y, Z) of the object point.

$\begin{matrix}{\varphi = {\tan^{- 1}\left( \frac{Y}{X} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 182} \right\rbrack \\{\theta = {\frac{\pi}{2} - {\tan^{- 1}\left( \frac{Z}{\sqrt{X^{2} + Y^{2}}} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 183} \right\rbrack\end{matrix}$

Furthermore, referring to FIG. 55, the zenith angle θ and the elevationangle μ of the incident ray satisfy the relation given in Eq. 184.

$\begin{matrix}{\mu = {\frac{\pi}{2} - \theta - \chi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 184} \right\rbrack\end{matrix}$

FIG. 56 is a conceptual drawing of an uncorrected image plane accordingto the eighth embodiment of the present invention, and FIG. 57 is aconceptual drawing of a processed image plane. As has been marked inFIG. 57, the lateral dimension of the processed image plane is W, andthe longitudinal dimension is H. Unlike in the examples of the previousembodiments, the reference point O″ on the processed image plane doesnot correspond to the intersection point between the optical axis andthe image sensor plane, but corresponds to the intersection pointbetween the said small circle(5239) and the X-Z plane (i.e., thereference plane). The coordinate of the said reference point is (x″_(o),y″_(o)). Furthermore, the lateral distance from the said reference pointto the third point P″ on the processed image plane is proportional tothe azimuth angle of the said object point(5504), and the longitudinaldistance is proportional to a monotonic function G(μ) of the elevationangle passing through the origin. If the longitudinal distance in theprocessed image plane is made proportional to the longitudinal distancein the object plane, then the said monotonic function is given by Eq.185.

G(μ)=tan μ  [Math Figure 185]

Eq. 185 has the same geometrical meaning as in the first embodiment ofthe present invention.

The distance from the said tangential point(5509) to the optical axis,in other words, the axial radius(5537), is given as Scos_(χ). In thisembodiment, this distance is considered as the radius of the objectplane. Therefore, the lateral dimension of the object plane must satisfythe following Eq. 186, where c is proportionality constant.

2πS cos χ=cW  [Math Figure 186]

Furthermore, considering the range of the said elevation angle, thefollowing relation given in Eq. 187 must holds.

$\begin{matrix}{{{{SG}(\mu)} = {c\left( {y^{''} - y_{o}^{''}} \right)}}{{Therefore},{{{Eq}.\mspace{11mu} 188}\mspace{14mu} {must}\mspace{14mu} {be}\mspace{14mu} {{satisfied}.}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 187} \right\rbrack \\{{B \equiv \frac{S}{c}} = {\frac{W}{2\; \pi \; \cos \; \chi} = \frac{y^{''} - y_{o}^{''}}{G(\mu)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 188} \right\rbrack\end{matrix}$

Here, B is a constant. On the other hand, another monotonic functionF(μ) is defined as in Eq. 189.

$\begin{matrix}{{F(\mu)} \equiv \frac{G(\mu)}{\cos \; \chi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 189} \right\rbrack\end{matrix}$

Therefore, Eq. 190 can be obtained from Eqs. 188 and 189.

$\begin{matrix}{{A \equiv {B\; \cos \; \chi}} = {\frac{W}{2\; \pi} = \frac{y^{''} - y_{o}^{''}}{F(\mu)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 190} \right\rbrack\end{matrix}$

The following equation can be obtained from Eqs. 185, 189 and 190.

$\begin{matrix}{{F(\mu)} = {\frac{\tan \; \mu}{\cos \; \chi} = \frac{y^{''} - y_{o}^{''}}{A}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 191} \right\rbrack\end{matrix}$

Therefore, the elevation angle corresponding to the said third point isgiven by Eq. 192.

$\begin{matrix}{\mu = {{F^{- 1}\left( \frac{y^{''} - y_{o}^{''}}{A} \right)} = {\tan^{- 1}\left\{ {\frac{\cos \; \chi}{A}\left( {y^{''} - y_{o}^{''}} \right)} \right\}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 192} \right\rbrack\end{matrix}$

Furthermore, the zenith angle can be easily obtained from the elevationangle of the incident ray.

$\begin{matrix}{\theta = {\frac{\pi}{2} - \mu - \chi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 193} \right\rbrack\end{matrix}$

On the other hand, the azimuth angle corresponding to the said thirdpoint is given by Eq. 194.

$\begin{matrix}{\varphi = {\frac{2\; \pi}{W}\left( {x^{''} - x_{o}^{''}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 194} \right\rbrack\end{matrix}$

Therefore, using equations 190 through 194, the zenith and the azimuthangles of the incident ray corresponding to the third point on theprocessed image plane can be obtained, and using this, image processingcan be done as in the previous embodiments.

Identical to the previous examples, considering the fact that all theimage sensors and display devices are digital devices, image processingprocedure must use the following set of equations. First, the desirablesize (I_(max), J_(max)) of the processed image plane, the location(I_(o), J_(o)) of the reference point, and the vertex half-angle of thecone, in other words, the reference angle χ, are set-up. In thisembodiment, the reference angle χ takes a value that is larger than −90°and smaller than 90°. Then, the said constant A is given by Eq. 195.

$\begin{matrix}{A = \frac{J_{\max} - 1}{2\; \pi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 195} \right\rbrack\end{matrix}$

By using J_(max)−1 as the numerator as in this case, the first column(i.e., the column with J=1) of the processed image plane exhibits thesame information as the last column (i.e., the column with J=J_(max)).In a panoramic image exhibiting the view of 360° directions, it appearsnatural when the left edge and the right edge matches. However, if sucha duplicate display of information is not preferred, then it is onlynecessary to change the numerator in Eq. 195 as J_(max). Next, for allthe pixels (I, J) on the said processed image plane, the elevation angleμ₁ and the azimuth angle φ_(J) given by Eq. 196 and 197 are computed.

$\begin{matrix}{\mu_{I} = {{F^{- 1}\left( \frac{I - I_{o}}{A} \right)} = {\tan^{- 1}\left\{ {\frac{\cos \; \chi}{A}\left( {I - I_{o}} \right)} \right\}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 196} \right\rbrack \\{\varphi_{J} = {\frac{2\; \pi}{J_{\max} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 197} \right\rbrack\end{matrix}$

On the other hand, the zenith angle of the incident ray is given by Eq.198.

$\begin{matrix}{\theta_{I} = {\frac{\pi}{2} - \chi - \mu_{I}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 198} \right\rbrack\end{matrix}$

The image height r₁ on the image sensor plane is obtained using Eq. 199.

r ₁ =r(θ₁)  [Math Figure 199]

Then, the position (K_(o), L_(o)) of the second intersection point onthe uncorrected image plane and the magnification ratio g are used tofind the position of the second point on the uncorrected image plane.

x′ _(I,J) =L _(o) +gr _(I) cos φ_(J)  [Math Figure 200]

y′ _(I,J) =K _(o) +gr _(I) sin φ_(J)  [Math Figure 201]

Once the position of the corresponding second point has been found, theninterpolation methods such as described in the first through the thirdembodiments can be used to obtain a panoramic image. Such a processedimage plane satisfies the relation given in Eq. 202 or Eq. 203.

$\begin{matrix}{\frac{J_{\max} - 1}{2\; \pi} = \frac{I_{\max} - 1}{{F\left( \mu_{Imax} \right)} - {F\left( \mu_{1} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 202} \right\rbrack \\{\frac{J_{\max}}{2\; \pi} = \frac{I_{\max} - 1}{{F\left( \mu_{Imax} \right)} - {F\left( \mu_{1} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 203} \right\rbrack\end{matrix}$

FIG. 58 is a panoramic image extracted from FIG. 26 using this method,whereof the lateral dimension of the processed image plane isJ_(max)=720 pixels, the longitudinal dimension is I_(max)=120 pixels,the position of the reference point is (I_(o)=1, J_(o)=1), and thereference angle is χ=0. On the other hand, in FIG. 59, the position ofthe reference point is (I_(o)=60.5, J_(o)=1), the reference angle isχ=45°, and the function given in Eq. 185 has been used. From the FIGS.58 and 59, it is clear that a natural looking panorama can be obtained.

Ninth Embodiment

FIG. 60 is a conceptual drawing of an object plane according to theninth embodiment of the present invention. The main difference betweenthe examples in the eighth and the ninth embodiments of the presentinvention is the fact that, a cone contacting the celestial sphere isused as the object plane in the eighth embodiment, while the celestialsphere itself is used as the object plane in the ninth embodiment. Allthe other aspects are mostly identical. In the ninth embodiment, also, acelestial sphere(6030) with a radius S is assumed that takes the nodalpoint N of the lens as the center of the sphere. When all the pointshaving a latitude angle χ from the ground plane, in other words, the X-Yplane, are marked, then the collection of these points form a smallcircle on the celestial sphere. The elevation angle μ of the incidentray is measured with respect to this small circle, and the zenith angleθ and the elevation angle μ of the incident ray satisfy the followingrelation given in Eq. 204.

$\begin{matrix}{\mu = {\frac{\pi}{2} - \theta - \chi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 204} \right\rbrack\end{matrix}$

The conceptual drawing of the uncorrected image plane according to theninth embodiment of the present invention is identical to FIG. 56, andthe conceptual drawing of the processed image plane is identical to FIG.57. As has been marked in FIG. 57, the lateral dimension of theprocessed image plane is W, and the longitudinal dimension is H. Thereference point O″ on the processed image plane corresponds to theintersection point between the said small circle and the X-Z plane(i.e., the reference plane). The coordinate of the said reference pointis (x″_(o), y″_(o)). Furthermore, the lateral distance from the saidreference point to the third point P″ on the processed image plane isproportional to the azimuth angle of the said object point(6004), andthe longitudinal distance is proportional to an arbitrary monotonicfunction F(μ) of the elevation angle from the said tangentialpoint(6009) to the said object point(6004), where the function ispassing through the origin. Similar to the second embodiment of thepresent invention, the said monotonic function can be given by Eqs. 205or 206.

F(μ)=μ  [Math Figure 205]

$\begin{matrix}{{F(\mu)} = {\ln \left\{ {\tan \left( {\frac{\mu}{2} + \frac{\pi}{4}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 206} \right\rbrack\end{matrix}$

The radius of the object plane of the present embodiment is taken as theradius of the celestial sphere. Therefore, the lateral dimension of theobject plane must satisfy the relation given in Eq. 207, where c isproportionality constant.

2πS=cW  [Math Figure 207]

Furthermore, considering the range of the elevation angle, the relationgiven in Eq. 208 must be satisfied.

SF(μ)=c(y″−y″ _(o))  [Math Figure 208]

Therefore, the relation given in Eq. 209 must hold true.

$\begin{matrix}{A = {\frac{S}{c} = {\frac{W}{2\pi} = {\frac{y^{''} - y_{o}^{''}}{F(\mu)} = \frac{H}{{F\left( \mu_{2} \right)} - {F\left( \mu_{1} \right)}}}}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 209} \right\rbrack\end{matrix}$

Therefore, the elevation angle corresponding to the said third point isgiven by Eq. 210.

$\begin{matrix}{\mu = {F^{- 1}\left( \frac{y^{''} - y_{o}^{''}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 210} \right\rbrack\end{matrix}$

Furthermore, the zenith angle can be easily obtained from the elevationangle of the incident ray.

$\begin{matrix}{\theta = {\frac{\pi}{2} - \mu - \chi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 211} \right\rbrack\end{matrix}$

In this embodiment, the reference angle χ takes a value that is largerthan −90° and smaller than 90°. On the other hand, the azimuth anglecorresponding to the third point is given by Eq. 212.

$\begin{matrix}{\varphi = {\frac{2\pi}{W}\left( {x^{''} - x_{o}^{''}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 212} \right\rbrack\end{matrix}$

Therefore, using equations 208 through 212, the zenith and the azimuthangles of the incident ray corresponding to the third point on theprocessed image plane can be obtained, and using this, image processingcan be done as in the previous embodiments.

Identical to the previous examples, considering the fact that all theimage sensors and display devices are digital devices, image processingprocedure must use the following set of equations. First, the desirablesize (I_(max), J_(max)) of the processed image plane, the location(I_(o), J_(o)) of the reference point, and the reference angle χ areset-up. Then, the said constant A is given either by Eq. 213 or 214.

$\begin{matrix}{A = \frac{J_{\max} - 1}{2\pi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 213} \right\rbrack \\{A = \frac{J_{\max}}{2\pi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 214} \right\rbrack\end{matrix}$

By using J_(max)−1 as the numerator as in Eq. 213, the first column ofthe processed image plane exhibits the same information as the lastcolumn. On the other hand, by using J_(max) as the numerator as in Eq.214, all the columns correspond to different azimuth angles. Next, forall the pixels (I, J) on the said processed image plane, the elevationangle μ₁ and the azimuth angle φ_(J) given by Eq. 215 and 216 arecomputed.

$\begin{matrix}{\mu_{I} = {F^{- 1}\left( \frac{I - I_{o}}{A} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 215} \right\rbrack \\{\varphi_{J} = {\frac{2\pi}{J_{\max} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 216} \right\rbrack\end{matrix}$

On the other hand, the zenith angle of the incident ray is given by Eq.217.

$\begin{matrix}{\theta_{I} = {\frac{\pi}{2} - \chi - \mu_{I}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 217} \right\rbrack\end{matrix}$

The image height r₁ on the image sensor plane is calculated using Eq.218.

r ₁ =r(θ₁)  [Math Figure 218]

Next, the position (K_(o), L_(o)) of the second intersection point onthe uncorrected image plane and the magnification ratio g are used tofind the position of the second point on the uncorrected image plane.

x′ _(I,J) =L _(o) +gr _(I) cos φ_(J)  [Math Figure 219]

y′ _(I,J) =K _(o) +gr _(I) sin φ_(J)  [Math Figure 220]

Once the position of the corresponding second point has been found,interpolation methods such as described in the first and the secondembodiments can be used to obtain a panoramic image. Such a processedimage plane satisfies the relation given in Eq. 221 or Eq. 222.

$\begin{matrix}{\frac{J_{\max} - 1}{2\pi} = \frac{I_{\max} - 1}{{F\left( \mu_{1\; \max} \right)} - {F\left( \mu_{1} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 221} \right\rbrack \\{\frac{J_{\max}}{2\pi} = \frac{I_{\max} - 1}{{F\left( \mu_{1\; \max} \right)} - {F\left( \mu_{1} \right)}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 222} \right\rbrack\end{matrix}$

The desirable examples of the monotonically increasing function F(μ) ofthe elevation angle in this embodiment can be given by Eqs. 97, 99 and101. FIG. 61 is a panoramic image acquired using this method, and anequidistance projection scheme given in Eq. 99 has been used. Thelateral dimension of the processed image plane is J_(max)=720 pixels,the longitudinal dimension is I_(max)=180 pixels, the position of thereference point is (I_(o)=90.5, J_(o)=1), and the reference angle, inother words, the vertex half-angle of the cone, is χ=45°. On the otherhand, a Mercator projection scheme given in Eq. 101 has been used inFIG. 62. Especially, since the vertical field of view in FIG. 62 is 90°,an imaging system completely free from a dead zone in securitymonitoring can be realized.

Tenth Embodiment

The most of the imaging systems in the present invention excluding thosefrom the fifth and the sixth embodiments share many common features.FIG. 63 is a conceptual drawing of the world coordinate system that arecommon to the most of the embodiments of the present invention. For thesimplicity of argument, this will be referred to as the tenthembodiment. The tenth embodiment of the present invention is a devicethat includes an image acquisition means, an image processing means andan image display means, where the image acquisition means is a cameramounted with a wide-angle imaging lens rotationally symmetric about anoptical axis, the said image processing means extracts a panoramic imageby image processing the distorted wide-angle image acquired using thesaid image acquisition means, and the said image display means displaythe said panoramic image on a rectangular screen.

The world coordinate system of the tenth embodiment of the presentinvention takes the nodal point N of the said lens as the origin, and avertical line passing through the origin as the Y-axis. Here, a verticalline is a line perpendicular to the ground plane, or more precisely tothe horizontal plane(6317). The X-axis and the Z-axis of the worldcoordinate system are contained within the ground plane. The opticalaxis(6301) of the said wide-angle lens generally does not coincide withthe Y-axis, and can be contained within the ground plane (i.e., parallelto the ground plane), or is not contained within the ground plane.Herein, the plane(6304) containing both the said Y-axis and the saidoptical axis(6301) is referred to as the reference plane. Theintersection line(6302) between this reference plane(6304) and theground plane(6317) coincides with the Z-axis of the world coordinatesystem. On the other hand, an incident ray(6305) originating from anobject point Q having a rectangular coordinate (X, Y, Z) in the worldcoordinate system has an altitude angle δ from the ground plane, and anazimuth angle ψ with respect to the reference plane. The planecontaining both the said Y-axis and the said incident ray is theincidence plane(6306). The horizontal incidence angle ψ of the saidincident ray with respect to the said reference plane is given by Eq.223.

$\begin{matrix}{\psi = {{\tan^{- 1}\left( \frac{X}{Z} \right)}.}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 223} \right\rbrack\end{matrix}$

On the other hand, the vertical incidence angle (i.e., the altitudeangle) δ subtended by the said incident ray and the X-Z plane is givenby Eq. 224.

$\begin{matrix}{\delta = {\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}} \right)}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 224} \right\rbrack\end{matrix}$

Said elevation angle μ of the incident ray is given by Eq. 225, wherethe reference angle χ takes a value that is larger than −90°, andsmaller than 90°.

μ=δ−χ  [Math Figure 225]

On the other hand, if we assume the coordinate of an image point on ascreen corresponding to an object point with a coordinate (X, Y, Z) inthe world coordinate system is (x″, y″), then the said image processingmeans process the image so that the image point corresponding to anincident ray originating from the said object point appears on the saidscreen at the coordinate (x″, y″), wherein the lateral coordinate x″ ofthe image point is given by Eq. 226.

x″=cψ  [Math Figure 226]

Here, c is proportionality constant. Furthermore, the longitudinalcoordinate y″ of the said image point is given by Eq. 227.

y″=cF(μ)  [Math Figure 227]

Here, F(μ) is a monotonically increasing function passing through theorigin. In mathematical terminology, it means that Eqs. 228 and 229 aresatisfied.

$\begin{matrix}{{F(0)} = 0} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 228} \right\rbrack \\{\frac{\partial{F(\mu)}}{\partial\mu} > 0} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 229} \right\rbrack\end{matrix}$

The above function F can take an arbitrary form, but the most desirableforms are given by Eqs. 230 through 232.

$\begin{matrix}{{F(\mu)} = \frac{\tan \; \mu}{\cos \; \chi}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 230} \right\rbrack \\{{F(\mu)} = \mu} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 231} \right\rbrack \\{{F(\mu)} = {\ln \left\{ {\tan \left( {\frac{\mu}{2} + \frac{\pi}{4}} \right)} \right\}}} & \left\lbrack {{Math}\mspace{14mu} {Figure}\mspace{14mu} 232} \right\rbrack\end{matrix}$

The specification of the present invention is implicitly described withreference to the visible wavelength range, but the projection scheme ofthe present invention can be described by the same equations asdescribed above even in the millimeter and the microwave wavelengthranges, in the ultra violet wavelength range, in the near infraredwavelength range, in the far infrared wavelength range, as well as inthe visible wavelength range. Accordingly, the present invention is notlimited to imaging systems operating in the visible wavelength range.

Preferred embodiments of the current invention have been described indetail referring to the accompanied drawings. However, the detaileddescription and the embodiments of the current invention are purely forillustrate purpose, and it will be apparent to those skilled in the artthat variations and modifications are possible without deviating fromthe sprits and the scopes of the present invention.

INDUSTRIAL APPLICABILITY

According to the present invention, mathematically precise imageprocessing methods of extracting panoramic images that appear mostnatural to the naked eye from an image acquired using a camera equippedwith a wide-angle lens rotationally symmetric about an optical axis, anddevices using the methods are provided.

SEQUENCE LIST TEXT

panorama, distortion, projection scheme, equidistance projection,rectilinear projection, fisheye lens, panoramic lens, image processing,image correction, rear view camera, video phone

1. A panoramic imaging system comprising: an image acquisition means foracquiring a wide-angle image using a wide-angle imaging lensrotationally symmetric about an optical axis; an image processing meansfor generating a panoramic image based on the said wide-angle image; andan image display means for displaying the said panoramic image on arectangular screen, wherein; a coordinate of an image point on therectangular screen corresponding to an object point having a coordinate(X, Y, Z) in a world coordinate system, which has a nodal point of thewide-angle lens as an origin and a vertical line passing through theorigin as the Y-axis and an intersection line between a reference planecontaining the said Y-axis and the said optical axis of the lens and ahorizontal plane perpendicular to the said vertical line as the Z-axis,is given as (x″, y″), a horizontal incidence angle ψ, which an incidentray originating from the said object point makes with the said referenceplane, is given as ${\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}},$ avertical incidence angle δ, which the said incident ray makes with theX-Z plane, is given as${\delta = {\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}} \right)}},$an elevation angle μ of the said incident ray is given as μ=δ−χ, herein,χ is a reference angle larger than −90° and smaller than 90°, a lateralcoordinate x″ of the said image point is given as x″=cψ, herein, c isproportionality constant, a longitudinal coordinate y″ of the said imagepoint is given as y″=cF(μ), herein, the said function F(μ) is amonotonically increasing function passing through the origin, whichsatisfies the relations given as F(0)=0 and$\frac{\partial{F(\mu)}}{\partial\mu} > 0.$
 2. The panoramic imagingsystem of claim 1, wherein the function F(μ) of the elevation angle μ ofthe said incident ray is given by${F(\mu)} = {\frac{\tan \; \mu}{\cos \; \chi}.}$
 3. The panoramicimaging system of claim 1, wherein the function F(μ) of the elevationangle μ of the said incident ray is given by F(μ)=μ.
 4. The panoramicimaging system of claim 1, wherein the function F(μ) of the elevationangle μ of the said incident ray is given by${F(\mu)} = {\ln {\left\{ {\tan \left( {\frac{\mu}{2} + \frac{\pi}{4}} \right)} \right\}.}}$5. A method of obtaining a panoramic image, the method comprising:acquiring an uncorrected image plane using a camera equipped with arotationally symmetric wide-angle lens, where an optical axis of thesaid camera and a lateral side of an image sensor plane of the cameraare made parallel to the ground plane; and extracting a processed imageplane based on the said uncorrected image plane, wherein, the saiduncorrected image plane is a two dimensional array with K_(max) rows andL_(max) columns, a pixel coordinate of the optical axis on theuncorrected image plane is (K_(o), L_(o)), a real projection scheme ofthe said lens is an image height r obtained as a function of a zenithangle θ of a corresponding incident ray and given as r=r(θ), amagnification ratio g of the said camera is given as${g = \frac{r^{\prime}}{r}},$ wherein r′ is a pixel distance on theuncorrected image plane corresponding to the image height r, the saidprocessed image plane is a two dimensional array with I_(max) rows andJ_(max) columns, a pixel coordinate of the optical axis on the processedimage plane is (I_(o), J_(o)), a horizontal incidence angleψ_(I,J)≡ψ(I,J)=ψ_(J) of the incident ray corresponding to a pixel on thesaid processed image plane having a coordinate (I, J) is given as a solefunction of the said pixel coordinate J as${\psi_{J} = {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {J - J_{o}} \right)}},$herein, ψ₁ is a horizontal incidence angle corresponding to J=1,ψ_(Jmax) is a horizontal incidence angle corresponding to J=J_(max), avertical incidence angle δ_(I,J)≡δ(I,J)=δ_(I) of the said incident rayis given as a sole function of the said pixel coordinate I as${\delta_{I} = {F^{- 1}\left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}}},$herein, F⁻¹ is an inverse function of a continuous and monotonicallyincreasing function F(δ) of the vertical incidence angle δ, the signalvalue of a pixel having a coordinate (I, J) on the said processed imageplane is given by a signal value of a virtual pixel on the uncorrectedimage plane having a coordinate (x′_(I,J), y′_(I,J)), the coordinate(x′_(I,J),y′_(I,J)) of the said virtual pixel is given by a followingset of equations θ_(I, J) = cos⁻¹(cos  δ_(I)cos  ψ_(J))$\varphi_{I,J} = {\tan^{- 1}\left( \frac{\tan \; \delta_{I}}{\sin \; \psi_{J}} \right)}$r_(I, J) = r(θ_(I, J)) x_(I, J)^(′) = L_(o) + gr_(I, J)cos  φ_(I, J)y_(I, J)^(′) = K_(o) + gr_(I, J)sin  φ_(I, J).
 6. The method ofobtaining a panoramic image of claim 5, wherein the said function F isgiven as F(δ)=tan δ, and the said vertical incidence angle is given as$\delta_{I} = {\tan^{- 1}{\left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}.}}$7. The method of obtaining a panoramic image of claim 5, wherein thesaid function F is given as F(δ)=δ, and the said vertical incidenceangle is given as$\delta_{I} = {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}{\left( {I - I_{o}} \right).}}$8. The method of obtaining a panoramic image of claim 5, wherein thesaid function F is given as${{F(\delta)} = {\ln \left\{ {\tan \left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}}},$and the said vertical incidence angle is given as$\delta_{I} = {{2{\tan^{- 1}\left\lbrack {\exp \left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}} \right\rbrack}} - {\frac{\pi}{2}.}}$9. A method of obtaining a panoramic image, the method comprising:acquiring an uncorrected image plane using a camera equipped with arotationally symmetric wide-angle lens, where an optical axis of thecamera is made parallel to the ground plane; and extracting a processedimage plane based on the said uncorrected image plane, wherein, an anglebetween a lateral side of an image sensor plane of the said camera andthe ground plane is γ, the uncorrected image plane is a two dimensionalarray with K_(max) rows and L_(max) columns, a pixel coordinate of theoptical axis on the uncorrected image plane is (K_(o), L_(o)), a realprojection scheme of the said lens is an image height r obtained as afunction of a zenith angle θ of a corresponding incident ray and givenas r=r(θ), a magnification ratio g of the said camera is given as${g = \frac{r^{\prime}}{r}},$ wherein r′ is a pixel distance on theuncorrected image plane corresponding to the image height r, the saidprocessed image plane is a two dimensional array with I_(max) rows andJ_(max) columns, a pixel coordinate of the optical axis on the saidprocessed image plane is (I_(o), J_(o)), a horizontal incidence angle ofan incident ray corresponding to a pixel on the said processed imageplane having a coordinate (I, J) is given as${\psi_{I,J} = {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left\{ {{\left( {J - J_{o}} \right)\cos \; \gamma} + {\left( {I - I_{o}} \right)\sin \; \gamma}} \right\}}},$herein, ψ₁ is a horizontal incidence angle corresponding to J=1, andψ_(Jmax) is a horizontal incidence angle corresponding to J=J_(max), avertical incidence angle of the said incident ray is given as${\delta_{I,J} = {F^{- 1}\left\lbrack {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left\{ {{{- \left( {J - J_{o}} \right)}\sin \; \gamma} + {\left( {I - I_{o}} \right)\cos \; \gamma}} \right\}} \right\rbrack}},$herein, F⁻¹ is an inverse function of a continuous and monotonicallyincreasing function F(δ) of the vertical incidence angle δ, a signalvalue of a pixel having a coordinate (I, J) on the said processed imageplane is given by a signal value of a virtual pixel on the uncorrectedimage plane having a coordinate) (x′_(I,J),y′_(I,J)), the saidcoordinate (x′_(I,J),y′_(I,J)) of the virtual pixel is given by afollowing set of equationsθ_(I, J) = cos⁻¹(cos  δ_(I, J)cos  ψ_(I, J))$\varphi_{I,J} = {\tan^{- 1}\left( \frac{\tan \; \delta_{I,J}}{\sin \; \psi_{I,J}} \right)}$r_(I, J) = r(θ_(I, J))x_(I, J)^(′) = L_(o) + gr_(I, J)cos (γ + φ_(I, J))y_(I, J)^(′) = K_(o) + gr_(I, J)sin (γ + φ_(I, J)).
 10. The method ofobtaining a panoramic image of claim 9, wherein the said function F isgiven as F(δ)=tan δ.
 11. The method of obtaining a panoramic image ofclaim 9, wherein the said function F is given as${F(\delta)} = {\ln \; {\left\{ {\tan \left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}.}}$12. A method of obtaining a panoramic image, the method comprising:acquiring an uncorrected image plane using a camera equipped with arotationally symmetric wide-angle lens; and extracting a processed imageplane based on the said uncorrected image plane, wherein, an anglebetween the said camera optical axis and the ground plane is α, an anglebetween a lateral side of an image sensor plane of the camera and theground plane is γ, the uncorrected image plane is a two dimensionalarray with K_(max) rows and L_(max) columns, a pixel coordinate of theoptical axis on the uncorrected image plane is (K_(o), L_(o)), a realprojection scheme of the said lens is an image height r obtained as afunction of a zenith angle θ of the corresponding incident ray and givenas r=r(θ), a magnification ratio g of the said camera is given as${g = \frac{r^{\prime}}{r}},$ wherein r′ is a pixel distance on theuncorrected image plane corresponding to the image height r, theprocessed image plane is a two dimensional array with I_(max) rows andJ_(max) columns, a pixel coordinate of the optical axis on the processedimage plane is (I_(o), J_(o)), a horizontal incidence angleψ_(I,J)≡ψ(I,J)=ψ_(J) of an incident ray corresponding to a pixel on thesaid processed image plane having a coordinate (I, J) is given as a solefunction of the said pixel coordinate J as${\psi_{J} = {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {J - J_{o}} \right)}},$herein, ψ₁ is a horizontal incidence angle corresponding to J=1,ψ_(Jmax) is a horizontal incidence angle corresponding to J=J_(max), avertical incidence angle δ_(I,J)≡δ(I,J)=δ_(I) of the said incident rayis given as a sole function of the said pixel coordinate I as${\delta_{I} = {F^{- 1}\left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}}},$herein, F⁻¹ is an inverse function of a continuous and monotonicallyincreasing function F(δ) of the vertical incidence angle δ, a signalvalue of a pixel having a coordinate (I, J) on the said processed imageplane is given by a signal value of a virtual pixel on the uncorrectedimage plane having a coordinate (x′_(I,J),y′_(I,J)), the said coordinate(x′_(I,J),y′_(I,J)) of the virtual pixel is given by a following set ofequations X_(I, J) = cos  δ_(I)sin  ψ_(J) Y_(I, J) = sin  δ_(I)Z_(I, J) = cos  δ_(I)cos  ψ_(J)X_(I, J)^(″) = X_(I, J)cos  γ + Y_(I, J)sin  γY_(I, J)^(″) = −X_(I, J)cos  α sin  γ + Y_(I, J)cos  α cos  γ + Z_(I, J)sin  αZ_(I, J)^(″) = X_(I, J)sin  α sin  γ − Y_(I, J)sin  α cos  γ + Z_(I, J)cos  αθ_(I, J) = cos⁻¹(Z_(I, J)^(″))$\varphi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}^{''}}{X_{I,J}^{''}} \right)}$r_(I, J) = r(θ_(I, J))x_(I, J)^(′) = L_(o) + gr_(I, J)cos  (φ_(I, J))y_(I, J)^(′) = K_(o) + gr_(I, J)sin (φ_(I, J)).
 13. The method ofobtaining a panoramic image of claim 12, wherein the said function F isgiven as F(δ)=tan δ, and the said vertical incidence angle is given as$\delta_{I} = {\tan^{- 1}{\left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}.}}$14. The method of obtaining a panoramic image of claim 12, wherein thesaid function F is given as${F(\delta)} = {\ln \; {\left\{ {\tan \left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}.}}$and the said vertical incidence angle is given as$\delta_{I} = {{2{\tan^{- 1}\left\lbrack {\exp \left\{ {\frac{\psi_{J\; \max} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}} \right\rbrack}} - {\frac{\pi}{2}.}}$15. A method of obtaining a panoramic image, the method comprising:acquiring an uncorrected image plane using a camera equipped with arotationally symmetric wide-angle lens, wherein the optical axis of thecamera is made perpendicular to the ground plane; and extracting aprocessed image plane based on the said uncorrected image plane,wherein, the uncorrected image plane is a two dimensional array withK_(max) rows and L_(max) columns, a pixel coordinate of the optical axison the uncorrected image plane is (K_(o), L_(o)), a real projectionscheme of the said lens is an image height r obtained as a function of azenith angle θ of a corresponding incident ray and given as r=r(θ), amagnification ratio g of the said camera is given as${g = \frac{r^{\prime}}{r}},$ wherein r′ is a pixel distance on theuncorrected image plane corresponding to the image height r, theprocessed image plane is a two dimensional array with I_(max) rows andJ_(max) columns, a pixel coordinate of the optical axis on the processedimage plane is (I_(o), J_(o)), an elevation angle of an incident raycorresponding to a pixel on the said processed image plane having acoordinate (I, J) is given as a sole function of the pixel coordinate Ias ${\mu_{I} = {F^{- 1}\left( \frac{I - I_{o}}{A} \right)}},$ herein, Ais a constant given as ${A = \frac{J_{\max} - 1}{2\pi}},$ F⁻¹ is aninverse function of a continuous and monotonically increasing functionF(μ) of the elevation angle μ, an azimuth angle of the said incident rayis given as a sole function of the said pixel coordinate J as${\varphi_{J} = {\frac{2\pi}{J_{\max} - 1}\left( {J - J_{o}} \right)}},$a signal value of a pixel having a coordinate (I, J) on the saidprocessed image plane is given by a signal value of a virtual pixel onthe uncorrected image plane having a coordinate (x′_(I,J),y′_(I,J)), thesaid coordinate (x′_(I,J),y′_(I,J)) of the virtual pixel is given by afollowing set of equationsθ₁=π/2−χ−μ₁r ₁ =r(θ₁)x′ _(I,J) =L _(o) +gr _(I) cos φ_(J)y′ _(I,J) =K _(o) +gr _(I) sin φ_(J), wherein χ is a reference anglelarger than −90° and smaller than 90°.
 16. The method of obtaining apanoramic image of claim 15, wherein the said function F is given as${{F(\mu)} = \frac{\tan \; \mu}{\cos \; \chi}},$ and the saidelevation angle is given as$\mu_{I} = {\tan^{- 1}{\left\{ {\frac{2\pi \; \cos \; \chi}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}.}}$17. The method of obtaining a panoramic image of claim 15, wherein thesaid function F is given as F(μ)=μ, and the said elevation angle isgiven as$\mu_{I} = {\frac{2\pi}{J_{\max} - 1}{\left( {I - I_{o}} \right).}}$18. The method of obtaining a panoramic image of claim 15, wherein thesaid function F is given as${{F(\mu)} = {\ln \left\{ {\tan \left( {\frac{\mu}{2} + \frac{\pi}{4}} \right)} \right\}}},$and the said elevation angle is given as$\mu_{I} = {{2\; {\tan^{- 1}\left\lbrack {\exp \left\{ {\frac{2\pi}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}} \right\rbrack}} - {\frac{\pi}{2}.}}$